REPORT ON THE ACTIVITIES OF THE EVALUATION PANEL September 2013
1. BACKGROUND The i2i competition was designed to reward creativity that has the potential to: a) Serve the national community by having positive development on Trinidad and Tobago economically, environmentally, or socially, and, b) Be commercially viable. The possibility of award of a grant of “up to $200,000” to finance the commercial development of the idea was the key factor in attracting submissions. In this context there is no single winner and each submission “stood on its own merit”, the limit being the extent of funding available through the Innovation Financing Facility (IFF). The Eligibility Criteria are included in Attachment 1, and the Evaluation Process and Criteria in Sections 4 and 5 of this Report. For each submission therefore, the Evaluation Panel focused its effort on testing the idea, and the potential implementation of the idea, as the two commonly accepted aspects of INNOVATION. The competition was launched on May 29, 2013, with a deadline for submissions of July 12, 2013. The Evaluation Panel was installed on July 12, by letter of July 10th, and the process was started on July 18th. It was completed on August 31st with fourteen (14) meetings being held for a total of approximately 69 hours.
2. TERMS OF REFERENCE The role of the Evaluation Panel was “To assess submitted proposals and project ideas based on a set of evaluation criteria in the respective area, and provide feedback on same in a marking framework”, and “To recommend to the Council for Competitiveness and Innovation (CCI) projects that should be awarded funding under the Innovation Financing Facility (IFF).
3. RESOURCES a. Evaluation Panel The Evaluation Panel comprised of: Mr Hayden Ferreira, Chairman Engineering Consultant Chairman - CARIRI Mr Robert Tang Yuk, Deputy Chairman Managing Director TYE Manufacturing Company Ltd Ms Rachel Renie Managing Director Market Movers Ltd. Ms Jeanelle Frontin Project Officer, Project Management Department, Central Bank Dr. Puran Bridgemohan Programme Professor Centre for Biosciences, Agriculture, and Food Technology - UTT Mr Armand Jackson Assistant Director, Socio Economic Policy Planning Division, Ministry of Planning & Sustainable Development Prof. Pathmanathan Umaharan Director, Cocoa Research Unit; Professor of Genetics – Faculty of Science & Agriculture, UWI Mr Sharaz Ahamad Director, Origination & Capital Markets Products, CIBC First Caribbean International Bank Dr Graham King Lecturer, Department of Mechanical Engineering, UWI b. Administrative Support Administrative support was provided by Mr Arvinda Rampersad, Ms Vikki Arjoon, and Ms. Latoya Charles, all of the Ministry of Planning and Sustainable Development.
4. EVALUATION PROCESS The process outlined and agreed at the first meeting of the Panel was: a. Receive and Register submissions This was done by the administrative staff prior to the Panel’s deliberations. In registration, a unique number (Tracking Code) was associated with each submission and this was used in processing, such that the Panel was not aware of the originator(s) of the submission. b. Compliance Review of submissions The administrative staff further reviewed submissions and removed those that were incomplete and/or otherwise non-compliant with the submission requirements for eligibility. In some cases, prior to the date of closing, the submissions were returned to the originator by staff for completion and re-submission. c. Pre-Screen of Submissions The initial i2i event in 2012 was overwhelmed with an unexpected number of 400 submissions, and this tested the Evaluation Panel’s ability to complete the process in reasonable time. With a higher number of submissions anticipated in 2013, it was determined that the introduction of a filtering process was an essential requirement. By doing this the number of submissions that came to the Evaluation Panel was limited to those with greater potential for earning an award, thereby making more efficient use of the Panel’s time. d. Evaluation – Stage 1: Scoring i. Target timeframe allotment 15 minutes per submission ii. Agree on CRITERIA iii. Agree on Meeting Quorum iv. 3-Stage process: Review – 5 mins, Individual activity, Share – 5 mins, Group activity, Rate – 3 mins, Individual activity, v. Collation by Competition Administration (Refer to Report Section 6 for Comments on the elements above.) e. Evaluation – Stage 2: Grant Assignments i. Review overall results to determine “cut-off” score point. ii. Select and review submissions exceeding cut-off point. iii. Consider each submission and determine the Grant to be awarded. In considering the Grant the following scope would be considered: Prototype development activity including design, testing, and field trials.
Market testing, if necessary to establish proof of concept. Expenditure on services and testing up to 40% of project cost. Raw materials necessary to achieve proof of concept. Equipment necessary to achieve proof of concept. IP investigation and documentation, but not patent application.
Specifically not considered, as guided by the Completion Rules, were: Purchase/rental of non-critical assets (e.g. buildings, vehicles and furniture). Construction of new infrastructure except to achieve proof of concept. Utilities. Travelling costs. Salary or allowances of any personnel employed by the applicant. f.
Special Consideration Though promoted as a “competition”, the event in its construction is more of a process for the selection of worthy ideas for grant funding for development to a point of potential commercial success. This milestone was defined by the Panel as the “proof of concept” point which was used at the Grant Assignment stage. The competition aspect was then from the perspective that only a limited number of the submissions would be expected to succeed to the final stage of assignment of a Grant. The Evaluation Process would therefore concern itself more with the selection of a number of worthy submissions than with a single winner of the competition, and with no prior limit having been placed on the number of worthy Awardees.
5. EVALUATION CRITERIA The Evaluation Panel discussed the proposed Evaluation Criteria (as published in the “Competition Details and Guidelines”) in context of the goals of the competition. Minor change to in the interpretation of two criteria (Items (h) and (i)) sought to better accommodate submissions in context of the 2012 experience. The original Criteria were: a. Innovativeness of the project submissions Project proposals must indicate whether it is recombination, fusion, integration, replication or refinement of existing technologies with improved value, enhanced efficiency or cost reduction. b. Societal benefits (only relevant to social initiatives) The project proposals must clearly describe the socio-economic benefits to the community. The proposals must also describe the community chosen and their involvement in the project.
c. Credibility of project proposal The project proposal must be clear, accurate and consistent with the objectives of the i2i competition and the Innovation Financing Facility. It must have achievable milestones and methodologies (to be in the timeline in the submission form) that can be completed within the agreed/given time frame. d. Appropriateness of methodology The applicant must provide sufficient information or evidence on the appropriateness of the chosen methodology (new or established methods/techniques). This should outline the sequence of proposed actions and identify these actions as numbered stages, steps and phases (explained in the justification details of the timeline). e. Appropriateness of milestones The proposed milestones must be appropriate and consistent with the project objectives and activities. f.
Competency of the Project Team or Individual Project teams should consist of qualified and technically competent members / institutions with respect to technical and commercialization aspects. Roles and responsibility of collaborators involved in the project should be clearly defined. Involvement of consultants in the project should be justified and with details submitted.
g. Commercialization Prospect (only applicable to relevant business ideas) The project should indicate clearly the expected outcome and demonstrate potential for commercialization. h. Financial Capability Applicants should express the ability to raise additional capital or indicate sources of additional capital to finance any portion of project cost not funded by the IFF. i.
Risk Mitigation The applicant must state the possible risks (technology risk, financial risk and time risk) that may affect the implementation or completion of the project, and steps likely to mitigate the risk.
6. COMMENTS FROM THE EVALUATION PROCESS a) Criteria Two Criteria “Financial Capability” and “Risk” initially gained the Panel’s attention as in the 2012 edition of the Competition these were the least well addressed Criteria in the submissions. One can certainly argue that an expectation of Financial Capability could prejudice the evolution of an idea seed, which would be counter to the intent of the Competition. The Evaluation Panel therefore preferred to re-define these two Criteria in terms of “Impact” (extent of economic stimulation) and “Scalability/Exportability”, which is self-explanatory.
Further to the above, the Criteria â€œSocietal Benefitsâ€? was taken to include Environmental benefits as well. b) Criteria Weights Consistent with the changes in the Criteria identified in (a) above, the Panel agreed to the following Criteria Weights. Criteria
Innovativeness of Project proposals Societal benefits Credibility of proposals Appropriateness of methodology Appropriateness of milestones Competency of the Project Team (individual) Commercialization prospect/ Economic Sustainability Impact Scalability/Exportability
20 10 15 10 5 15 15 5 5
c) Panel Meetings & Quorum A total of fourteen (14) meetings of the Evaluation Panel were held. Attachment 2 provides details of dates and durations. A quorum of five (5) was agreed by Panel Members to accommodate the times when some Members may not be available. The nine member Panel facilitated this. d) The Screening Process i. Administrative A total of four hundred and seventy-one (471) submissions were received at the closing. Of these, four (4) duplicate submissions were identified, and a further thirteen (13) were rejected as incomplete submissions. A total of four hundred and fifty-four (454) submissions were therefore processed. ii. Pre-Screening As indicated in Section 4(c), it was determined that the introduction of a prescreening process was an essential requirement in order to complete the Evaluation process in a reasonable timeframe. This pre-screening reviewed all 454 submissions according to the same Criteria used by the Evaluation Panel, and provided a rating on each. The Evaluation Panel then selected a number of submissions for its review. The Panel selected 181 submissions, comprised not only of the higher rated ones of the pre-screen,
but also several selected for high criteria scores (e.g. Innovation), and a few selected at random. In this manner, the pre-screening activity was not allowed to inadvertently bias the Evaluation Panel’s effort. The pre-screening was done by the University of Trinidad and Tobago (UTT), and set up by the Ministry’s i2i Administration using the same Confidentiality
undertakings as exist for the Evaluation Panel. The pre-screening activity was at all times guided by the Evaluation Panel, which maintained physical control of all entries. The pre-screening process eliminated applications which were largely business ideas that did not have a creative element, ideas that have already been taken beyond the proof-of-concept stage, or applications that were poorly put together. The UTT was selected for their entrepreneurial and innovation focus, with which is associated an appropriately high level of competent staff, and, as a University, their Administration routinely handles confidential material and processes. Eight (8) of their Faculty were involved in the pre-screen process. e) The Evaluation Process i. Review The Review step in the evaluation was an individual review by each Panel Member of the submission. ii. Share In the Share step, some questions or perspectives on clarity were shared among Members. It was agreed, and care was taken to avoid value judgements that would bias the thinking of other Members. The propensity for “Group Think” (convergence of a group’s perspective, sometimes being led by a single person) was the concern that was being avoided. iii. Rate Each submission was rated on individual evaluation sheets (i.e. one sheet per Member). The rating scale was 1-10, with 10 being the highest. Members calibrated themselves by agreeing that a rating of “5” represented an “average” assessment for the particular criteria. Ratings of 1-4 were therefore to be appropriately applied as Ratings of 6-10. This agreement circumvented a common human failing to avoid the application of ratings that are deemed “below average”. iv. Cut-off Point In context of the process used for the evaluation, it would not have been possible to pre-determine a “passing score”. Given the four hundred and fiftyfour (454) that passed the administrative screening and which the evaluation
process considered, the Panel’s rule-of-thumb expectation was that approximately 10-15 percent would have been of a satisfactory level to proceed to the Grant award stage. It was, however, expected that upon assessment of the scored data, a “natural break point” would be identified that would yield this number. The one caveat to the above was a final check that was made to ensure that no single recipient or company received more than one award. The Panel considered this a reasonable implementation of the intent of the Competition, even while encouraging multiple submissions from individuals and Companies. v. Grant Award In considering the amount of the Grant to be awarded, the Panel considered the nature of the idea/project, the reasonableness of the Plan outlined in the submission and associated costs, and, in particular, the elements of the cost that would take the project to the “proof of concept” point. The basic assumption here is that once the “proof of concept” point was achieved, the project could now be taken to a funding agency with a higher degree of confidence in achieving financial support to commercialise the idea. Alternatively the Awardee could patent the idea and licence it to a third party that may wish to take it to commercialisation. Consideration was, therefore, not given to operational needs as these would clearly be past the “proof of concept” point. One valid cost aspect that was deliberately not considered at this stage is associated with IP investigation and documentation. To have considered this would have required that the Panel make a determination on the likely status of the idea, or conduct at least a preliminary IP investigation at a time cost. A key aspect of the Grant Award stage was the conduct of internet searches of all submissions proposed for Award. Where similar ideas or projects were identified, the Panel engaged in a deliberation to achieve final agreement. f) Duration of Evaluations As was experienced in 2012, the Panel went through an initial Learning Curve in the Evaluation Process. The first few submissions evaluated required in excess of 20 minutes. Quickly, however, this reduced to an average of approximately 15 minutes, with a range of 12-18 minutes, as the Panel gained collective expertise. These times are a little higher than 2012, reflective of the value of the pre-screen process in bringing a higher quality of submission to the Panel. The value of having Evaluation Panel Members with last years’ experience, who again served on the Panel this year, was seen in the quick set-up of the Panel’s work.
g) Competence of the Evaluation Panel The Members of the Panel were selected for independence and professional credibility, and to have a mix of professions and experiences, including in both the public and private sectors. This mix of breadth and depth definitely facilitated discussions, significantly enhanced the evaluation process, and allowed all members to contribute to the process. Members were dedicated to the task, and, in spite of other personal/professional obligations, the Panel were able to meet regularly to complete the Terms of Reference. h) Trends from the Submissions i. Summary of Submissions As indicated earlier, a total of four hundred and seventy-one (471) submissions were made, the majority via the on-line facility. Of these four hundred and fifty-four (454) passed the administrative test for completeness. ii.
Completeness of Submissions In 2012 the level of incomplete entries was a high 20%. It therefore is a very positive trend that on this occasion that level was under 3%. This reasonably reflects keener interest by the potential population with innovative ideas. The other aspect of the Completeness issue is more about Appropriateness. Like in 2012, several submissions were simply a business plan/direction needing funding. They typically had no aspect of innovative idea and needed no research or development. These were basically submitted by individuals who believed this was a funding mechanism for their business plans. These submissions on this occasion numbered twenty-nine (29) and had to be set aside.
Work Programme Timeline & Application of Funds As experienced in 2012, a challenge for persons submitting entries seemed to be the Work Programme and associated Timeline and Costs, all part of a normal Business Plan. Reasonably, there seems to be a lack of even basic understanding of the development of simple Business Plans. On the positive side, this reality presents significant leaning opportunity for several of the Awardees to go through this process with their successful ideas.
Business Proposition or Sustainable Impact Associated with the deficiency identified in Section 6 h(iii) above is that of not having a specific requirement for a statement of the Business Proposition, and in particular addressing the question â€œhow will revenue be earned?â€?. Several submissions were guilty of this and required the Panel to exercise its judgement on the matter.
Comparative Distributions Attachment 3 presents the statistical results of the Submissions and Awardees. Evident here is a disappointing under-representation in Categories that one would expect high interest in, for example Agro-industrial Processing, Environment, Tourism, and Food and Beverage. On the other hand, the Panel also noted what it deems an over-representation by ICT related ideas. The 16% of Submissions that are recorded in Attachment 3 as ICT is actually understated as the Creative, and Services Categories have several submissions that could equally well have been classified as ICT. In seeking to understand the peculiarity, the Panel takes cognizance of the current popularity of the IT sector (inclusive of App development). The Panel, however, wonders whether some aspect of marketing or communication of the Competition may have inadvertently contributed to a bias towards particular potential interest groups or away from others, or alternatively a notion that innovation can only be considered in the context of ICT.
Robustness of Results Attachment 3 also provides evidence in the similarity between the two distributions in that the outcome (i.e. the list of Awardees) mirrors very well the distribution of the Submissions. This is noteworthy as each Submission was considered on its own merit, with no attempt to achieve a particular result in any Category. The results are therefore deemed to be robust. A further aspect of the robustness of the result is in the fact that each Awardee would have had the application reviewed four (4) times â€“ the initial screening for completeness, the pre-screening, the Evaluation Panel initial evaluation, the Evaluation Panel award granting evaluation.
Multiple Awards Though multiple submissions to the Competition are allowed, multiple Awards in one year to the same individual or Company is not. A confirmatory check on this was done by the Administrative Staff at the end of the Awards phase. Where there was an instance of more than one Award to the same person or group, the Evaluation Panel was advised and deliberated on which was the submission more likely to achieve the Proof of Concept stage in order to make a decision.
Repeat Submissions In making the checks of (vii) above, the Staff noted several persons who had entered the Competition in 2012, again made submissions in 2013, albeit on different ideas. The Panel views this as highly desirable and to be encouraged, even if it results in an individual achieving an Award in successive years, of course subject to successful completion of their Proof of Concept work scope in the prior year(s).
7. RECOMMENDATIONS FOR AWARD The Evaluation Panel makes a recommendation for basic Grants to fifty-three (53) submissions as identified in Attachment 4, for a total of $5,710,000, before VAT. If the maximum VAT becomes applicable, the Grant value is $6,566,500.
For and on behalf of the Evaluation Panel,
Hayden Ferreira Chairman, i2i Evaluation Panel
September 24, 2013
1. Applicants must be 18 years of age and over. 2. Individuals, teams or entities from the Diaspora, provided that their projects will be implemented in Trinidad and Tobago and/or provide jobs here. 3. In the case of companies, they must have a majority shareholding (at least 51%) held by nationals/ citizens of the Republic of Trinidad and Tobago including companies in the any in the Diaspora. 4. Applicants wishing to participate as Small and Medium-sized firms must have minimum asset base of $25,000 dedicated for this project. Start-up companies and teams are exempted from this stipulation but must provide justification and supporting documents on the ability to sustain their projects. 5. A work programme and timeline (submission guideline 7) must be also provided indicating the intended use of the funds and associated time periods for expenditure. 6. The project proposal should contain elements of technological and business innovation leading to commercialization of innovative products, processes and services. 7. The project team, partnership or entity should provide evidence of technical competency to undertake the proposed activity. 8. Application from an individual must be accompanied by a supporting letter from a referee i.e. from an independent person who has been associated with him/her and so qualified to indicate his/her ability to undertake the project. 9. Applicant must be a registered Cooperative or registered Government recognized Community Organization / Group, including non-governmental organisations in the Republic of Trinidad and Tobago 10. Applicant must show proof of financial capability that they can fund the portion of project costs not funded under the IFF. 11. Projects must be undertaken in the Republic of Trinidad and Tobago. 12. All successful participants must commit to a maximum of 30 hours of assistance after completing the process and implementing their projects towards successful enterprises or ventures to give back to future participants as mentors or coaches.
Evaluation Panel Meetings
1. Thursday July 18th 2013
2. Wednesday July 31st 2013
3. Saturday August 3rd 2013
4. Tuesday August 6th 2013
5. Thursday August
6. Tuesday August 13th 2013
7. Thursday August 15th 2013
8. Saturday August 17th 2013
9. Tuesday August 20th 2013
10.Thursday August 22nd 2013
11.Saturday August 24th 2013
13.Thursday August 29th 2013
14.Saturday August 31st 3013
Distribution of Submissions and Awardees among Categories L Awardees e No. % a 1 2%
Category Agro-industrial processing
Submissions No. % 15
Alternative/remedial energy and energy efficiency
Environment (clean technologies, eco-related activities)
Information and Communication Technology (ICT)
Manufacturing and manufacturing related
Primary agriculture, bio-technology
Food and Beverage
Bio-waste and other waste (including recycling activities)
Attachment 4 No. Tracking Number
Type of Submission
Alternative/Remedial Energy and Energy Efficiency
Increasing Engine Efficiency by Modifying the Turbo Charger
Martin Lee John
Movable Heads on an Integral Combustion Engine
Bim and Bam Adventure Series
The Calypso Song Book
Listen to your mom- 3D Animated Series
Environment (Clean Technologies, Eco-related Activities
Information and Communication Technology (ICT)
Crick-Crack Digital Game
Immersive Language Learning Application
Kevan St Brice
Caribbean Education Platform
Birds of Trinidad and Tobago Mobile App
Carlton De Silva Gordon
Passenger Seat Indicator System
Taxi Me to
Automative Sleep Imobiliser
Bluetooth key and lock proximity sensors
Construction of Solar Dryers for Sun Dried Banana Production
Greencrete Reinforced Concrete Composite
No. Tracking Number
Type of Submission
Manufacturing and Manufacturing Related
The Shop Wizard
A Multifunctional Mechanical Coupling/Locking System
GoBox the mobile kiosks
Mixed Flow Centrifugal Pump
Disposable sterilized Medical Rooms
Portable Automatic Water Closet
Wireless Paired Following Spotlight and Transmitter
Rythms & Hues, Sticks & Tones
See the Sound
Aaron Doug Deen
Multipurpose Angiographic Catheter
JW Venturi Acoustic Electronic Pan
Primary Agriculture, Bio-tchnology
Barley Fed Rabbits/Aeroponic Lettuce
The commercial production and sale of Arthospira platensis
No. Tracking Number
Type of Submission
Caribbean Market Square
Bertie Marshall Pan Institute
TT Creative Cloud
The City Kind: Urban Design & Architecture
Kids Programming Starter Program (KPSP)
The Spoken Syllabus
Customer Satisfaction Feedback
Peace Education Socialization Programme
Quick sell TT
Electronic Guided Tours of T&T
Food and Beverage
Caribbean Diabetic Cooking Show
The report generated by the independent Evaluation Panel for the idea2innovation 2013 competition hosted by the Council for Competitiveness...
Published on Oct 24, 2013
The report generated by the independent Evaluation Panel for the idea2innovation 2013 competition hosted by the Council for Competitiveness...