Timor-Leste Civil Society capacity assessment

Final Report of Baseline Organisational Capacity Assessment (BOCA)
EU Funded Project February 2012 – December 2013
 Prepared by
Tomas Freitas
Independent Evaluator
Dili, 10th of March 2014

Acronyms
AJEC                Associasaun Estudante Joventude Estacao Cajularan
CDC                 Centro Dezenvolvementu Comunitaria - Baucau
CDHD              Centru Feto Haburas Dezenvolvementu
CDP                 Centro Dezenvolvementu Popular – INURITIL
CDTL                Community Development Timor-Leste – Covalima
CFHD               Centro Feto Haburas Dezenvolvementu - Baucau
CFS                  Centro Formasaun de Servisu
CIACS               Centro Informasaun de Agricultura Cantina do Suco - Viqueque
CTI                   Community Transformation Institute - Baucau
CYC                  Children and Youth Centre – Ponta Leste, Baucau
DFW                Dili Film Work - Dili
DISIS                Dili Institute of Strategic and International Studies - Dili
EP                    Estrela Plus – Dili
ETADEP           Ema Mata Dalan Ba Progresio
FB                    Fundasaun BELUN – Baucau
FBHTL              Fundasaun Beneleba Haburas Timor-Leste - Liquica
FCBB                Foinsae Cdalak Buka Banati – Viqueque
FC                    Fundasaun CAILALO - Baucau
FECM               Fundasaun Educasaun Comunidade Matebian – Baucau
F-GRACA          Fundasaun Graca - Covalima
F-HABEER        Fundasaun HABEER - Bobonaro
F-HACSOLOK   Fundasaun Hacsolok - Bobonaro
F-HADER          Fundasaun Hadian Dezenvolvimentu EKonomia Rural - Liquica
F-HAMAHON   Fundasaun HAMAHON - Dili
FHM                Fundasaun Hadomi Malun – Covalima
FHTL                Fundasaun Haksolok Timor-Leste
FMF                 Fundasaun Moris Foun – Liquica
FMH                Fundasaun Moris Hamutuk – Liquica
F-MALAEDOI   Fundasaun MALAEDOI – Liquica
F-MANKLEDU  Fundasaun MANKLEDU - Liquica
F-NETIL            Fundasaun NETIL - Liquica
FINA                 Fitun Naraoman - Covalima
Loro Matan     Loron-Matan - Covalima
FOLSETIL         Fo Liman Serbi Timor-Leste - Liquica
FKSH                Feto Kbiit Servisu Hamutuk – Dili
F-KAEBAUK      Fundasaun KAEBAUK - Covalima
FRN                  Fundasaun ROMAN NUTETU – Covalima
FSS                   Fundasaun Sadan Saneti - Covalima
F-TOHA            Fundasaun Timor Oan Hamutuk - Viqueque
GHC                 Grupo Haburas Covalima
GFFTL              Grupo Feto Foin Sae Timor Lorosae – Dili
HAFOTI            Hamahon Feti Timor - Dili
HADEZTA         Haburas Dezenvolve Talento - Dili
HDTL               Haburas Dezenvolvementu Timor-Leste - Liquica
HDMTL            Halibur Deficiente Matan Timor Leste – Dili
HLT                  Hametin Lia Tatoli - Covalima
HIAM               Hamutuk Ita Ajuda Malu – Health – Dili
IBP                   Ita Ba Paz - Dili
IHL                   Institutu Luta Hamutuk – Dili
IT                     Info Timor - Baucau
JEF                   Juventude Esperanca ba Futuru – Covalima
JRB                  Juventude Rai Binan - Covalima
KDP                  Kolegas Das Pas – Viqueque
KDT                  Klibur Domin Tibar - Liquica
LABEH             Lalenok Ba Ema Hotu – Dili
LODA               Luta Organiza Dezenvolve Area - Liquica
KHC                 Knua Haberan Komunidade – Viqueque
MIPG - ET        Multi Interest Peace Group of Tasi Timor - Bobonaro
MOKO             Moris ho Comunidade - Viqueque
OHM               Organizasaun Haburas Moris – Bobonaro
PJF                   Prepara Juventude ba Futuru – Covalima
RHTO               Rae’es Hadomi Timor Oan – Dili
RCT                  Radio Comunidade Tokodede - Liquica
RPV                  Radio Povo Viqueuque
RN                   ROMAN NUTETU – Covalima
Sadan Saneti   Sadan Saneti - Covalima
TAHA               Tane Hamutuk - Liquica
TTAA                Technical Trainees Alumni Association Timor-Leste - Liquica                      
TECOMTIL       Technology Computer Timor-leste - Viqueque
UFD                 Unidade Feto ba Dezenvolvementu - Baucau

Acknowledgments
The author would like to thank the logistical and communications support provided by FONGTIL during this assessment. Special thanks to Tito Lopes, Cornelio do Rosario, Isaura Dias, Ricardo Ximenes, Natercia Mendes and Olivio Sequeira for their assistance during the interviews in Dili and also in the districts. The author would also like to thank the FONGTIL Director as well as the management team and all the staff who have contributed their time and knowledge during the evaluation process. The views expressed in this document in no way reflect the views of FONGTIL or the delegation of the European Commission in Timor-Leste. For further clarifications or additional information the author can be contacted on thomas_freitas@yahoo.com.

Executive Summary
The European Commission to Timor-Leste has funded the Baseline Organisational Capacity Assessment (BOCA), an assessment to measure the capacity development of its members, staff and Secretariat, as well as to ascertain what the agencies, donors and government think about the quality of the 267 FONGTIL members around the country. This last point the evaluator was unable to measure, as the baseline report has not yet been published. Regarding this, the evaluator has attempted to identify what has caused this delay and what factors have affected this activity. The evaluator has identified internal and external factors that have contributed to the failure of this activity.
​
This assessment has involved 67 member organisations, interviews with 96 people, including 39 directors, 16 program managers, 24 key staff, six DLO’s, and four board members, including a former board member, a director and one former director of FONGTIL and five MOCA members. This evaluation was conducted in six districts including Dili, and also involved a desk review, face to face interviews and focus group discussions as the methodology for evaluation.
Face to face interviews were used as the evaluation methodology in Dili, while focus group discussions were used in the districts; Viqueque, Baucau, Liquica, Bobonaro and Covalima.
This evaluation has come to the conclusion that there has been a failure of the MOCA project. Internal factors that have contributed to this failure are included in the 15 Activities Plan for 2013; however according to the evaluation and documents review, this included the fact that two activities could not be implemented due to limited time, and six main activities were requested for removal, meaning that then another two activities could not be executed. Due to a delay in re-entering in the database for MOCA assessment, this has caused a delay in analysing the baseline assessment and therefore the final report was also delayed for submission. The final result of the baseline assessment therefore could not be published, and because of that part of the BOCA assessment (point two) could not be done. 
External factors that have contributed to the failure of the MOCA project is the complexity of problems that its members face in their own organisation, which have accumulated for a long time and which have impacted on the organisation through, in some cases, an unwillingness to work to implement programs and activities. In this evaluation of 67 member organisations in six districts, 20% of them are still looking for donors, 8% do not care what FONGTIL is doing, 23% gave negative comments about FONGTIL, and 33% are still waiting for their problems to be fixed by FONGTIL. These attitudes have been accumulating for some time and have contributed to a souring of the relationship between the FONGTIL Secretariat and its member organisations.

Introduction
The Baseline Organisational Capacity Assessment (BOCA) was developed to evaluate another project called the Membership Organisational Capacity Assessment (MOCA). In accordance with the contract between FONGTIL and the European Commission, the MOCA project began in February 2012 and finished in December 2013. At the beginning of February 2014, FONGTIL recruited Tomas Freitas as the independent consultant to conduct the external evaluation. The evaluation was conducted over twenty (20) working days, with a total budget of US $5000. Both assessment tools are funded by the European Union based in Timor-Leste, with the aim of increasing capacity development among non-government organisations (NGOs) in Timor-Leste. This evaluation was part of the activities plan of the MOCA project, with a total budget of EUR192.966,45, divided  between the European Union grant totalling approximately EUR173.644,45 and from FONGTIL approximately EUR19.322,00 according to the contract (DCI-NSAPVD/2011/277-927).

Background of FONGTIL
As the umbrella organisation of local, national and international NGOs, civil society organisations and development partners in Timor-Leste, FONGTIL (Forum ONG Timor-Leste), according to the Annual Report of 2013, currently has 367 member organisations operational across the entire country. FONGTIL’s mandate is to do coordination, advocacy, and dissemination of information, and facilitate capacity development. These activities are integrated into FONGTIL’s strategic planning.  This includes approximately 323 local and national NGOs, 57 international NGOs, as well as seven candidates for local NGOs and five for international NGOs, all of which are under FONGTIL’s responsibility. FONGTIL has seven board members composed of five males and two females, six of whom are representatives of local and national organisations and one from an international NGO. As of 31 December 2013, FONGTIL had 33 permanent staff.
Non-governmental organisations have existed in Timor-Leste since the 1980s. During that time the work of NGOs was focused on emergency programs; in the 1990s some NGOs began to expand their programs, including community development. In the late 1990s, advocacy and human rights became central issues and the main program for NGOs in that period, and in 1998, fourteen organisations together took the initiative to establish Forum ONG Timor-Leste (FONGTIL).

Chronology of the MOCA assessment
One of FONGTIL’s mandates is to facilitate capacity development of its members. A questionnaire was developed to map the quality of organisations through identifying their strengths and weaknesses. The MOCA assessment project involved 267 samples, comprising of 195 NGOs, 52 Foundations, and 19 Associations. To run this assessment, FONGTIL employed 14 facilitators which included DLO’s and the MOCA team itself. The pilot testing in Dili began on the 9th of October and finished on the 30th of November 2012. It assessed 20 NGOs, and was conducted by seven members of the MOCA team. Before beginning the assessment, the facilitators were required to attend interviews training which was conducted over three days (29-31 January, 2013), the purpose of which was to guarantee the quality of the assessment. The assessment then began on the 1st of February 2013 in twelve districts, with a target of 80% of the 367 members, which totalled about 294 members. By the 22nd of July 2013, 267 had returned the questionnaire. The process of data entry began on 23rd of May and continued up to the 28th of August 2013, and took more than 97 days to complete.

This chronology only partially reflects the activities of the MOCA project; there are fifteen more activities which needed to be completed over four months, between September and December 2013. Some training was done for the capacity development of the members and staff of FONGTIL. The recruitment of an independent consultant for analysing the baseline database was delayed, and this affected the timing for the publication of the baseline assessment report.

Program rationale and logic
According to the Term of Reference (TOR) the objective of the BOCA is to assess the outcome of the MOCA project, which reviewed the progress of activities against the estimated results, and which targeted three main objectives as described below; 
1. Increased understanding of Member-Organizations capacity development needs, by themselves, their development partners/funders and FONGTIL, leading to better and realistic actions to address their priority needs, either by themselves and/or through the facilitation of FONGTIL, or with their partner donors, other International NGOs or key National NGOs.
2. Published results of the baseline assessment have effectively served as benchmark for FONGTIL-member organisations for future improvement and as a valuable reference for development partners in relation to human resource development in Timor-Leste.
3. FONGTIL Secretariat is effectively capacitated to efficiently and more effectively address its core service of providing sustainable organizational capacity development assistance to its members.
To evaluate these three main objectives, the evaluator used three different approaches as the methods for assessment, including data review and analysis, direct interviews and focus group discussions.
 
Methods of Evaluation
This evaluation was conducted by using the methodology of document review, face to face interviews and focus group discussions, as well as data analysis of the contract agreement between the European Commission and FONGTIL, the MOCA action plan, the interim and annual reports, as well as several related important documents. Details of these documents can be seen on the references list below. The evaluation began with two days of document review (10 – 11 Feb, 2014), followed by three days of face to face interviews (12, 13, 14 Feb, 2014) in Dili (five NGOs per day - fifteen in total). During the time in Dili, the evaluator also interviewed the MOCA teams. Questions included, what do they think about the MOCA assessment; what kind of benefits have they received from FONGTIL since they become members; and what kind of training do they want from FONGTIL that will help them to increase their capacity.

After Dili, the evaluation continued in the districts over five days, beginning in Viqueque, then Baucau, Liquica, Bobonaro and Covalima. In the districts the evaluators used a different approach for conducting the evaluation, that is focus group discussions rather than individual interviews. Due to limited time the discussion was conducted in one place and was organised by the FONGTIL District Liaison Officer. In Viqueque district, the meeting was attended by eleven NGOs (twelve were expected), in Baucau ten NGOs (ten were expected), in Liquica twelve members (fourteen were expected), in Bobonaro five members (of the expected fifteen). One of the reasons for the low attendance in Bobonaro was because it was raining heavily and the roads were slippery, making access to the location was difficult. Low attendance did mean however that those who did attend had a lot of time to express their views. In Suai the meeting was attended by fourteen of the expected eighteen participants. Details of participant attendance can be seen in the below table.   

The evaluation process in the districts consisted of three topics on the agenda; first, verification of key documents including the Constitution of the organisation, internal regulations, rules and guidelines for finance, human resources, assets, as well as monitoring, and evaluation reports. Secondly, evaluation of the relationship between the members and FONGTIL, which focused on capacity development, training and advocacy. Thirdly, the issue was raised by the members about the limits of donor support from external and internal sources, especially civil society funding from the Prime Minister’s Office. As described in the analytical report of the baseline assessment by an independent consultant, INSIGHT, 125 of 267 members rely on short term funding for specific projects, with a budget not exceeding $25,000. Therefore, the discussion about FONGTIL facilitating donor access to their members became an important topic of discussion.

The total number of individuals involved in this process of evaluation was 96 people, which included 39 directors, 16 program managers, 24 key staff, six DLO’s, and four board members, including a former board member, a director, one former director of FONGTIL and five MOCA members. The evaluation in the five districts (not Dili) involved interviews with the DLO’s, in order to determine the working relationship between the DLO’s and the Secretariat of FONGTIL in Dili.  After concluding the initial interviews in Dili and the other five districts, the interviews continued on in Dili, which included interviews with the DLO from Maubisse (responsible for Aileu and Ainaro districts). The interviews also involved the MOCA teams, some of team leaders, board members and the Executive Director of FONGTIL. The objective of these additional interviews in Dili was to clarify some issues that had been raised by the members and the DLO’s from districts.

The questions asked in the interviews revolved around the organisation having their own constitution, internal regulations, human resource guidelines, financial regulations, asset management guidelines, and guidelines for planning, monitoring, evaluation and reporting. The evaluator considered these questions to be key to identifying the capacity development of each member. The organisations were asked to share their experience of preparing and finalising these documents. It should be noted that many of the member organisations did not understand what the differences between some of these key documents.   

Strengths and weaknesses of the methodology
The strengths and weaknesses of this evaluation are varied; in Dili the face to face interviews meant that most of the time the evaluator met only with the Director him/herself, or just the program manager or even sometimes with just one staff member alone. This meant that it was difficult to verify whether what he or she said is the truth. During the interviews in Dili with fifteen organisations, only three organisations had more than one person participate in the interview. The majority of the time the person involved in the interview was the one who had filled out the MOCA questionnaire.

In the districts the focus group meetings had advantages and disadvantages; the advantage was that there was time to talk because the evaluator as the moderator gave opportunity to all of the participants to say something. But much of the time the men talked a lot compared to the women, and the women were not inclined to say anything contrary to what the men had said. For example in Baucau, when it came to a discussion about funding for civil society from government, the men expressed that they believed funding should be awarded based on the quality of the proposal, whereas the women preferred that all members had the same rights to receive funding, irrespective of the quality of the proposal. The discussion was not balanced due to the dominance of the men in the discussion. 

Findings                      
This report has the findings of the assessment into three categories, and sub-categories according to the terms of reference of the evaluation. First, to identify the capacity development of each member before and after they received training from FONGTIL or other organisation. Second, to evaluate the published results of the baseline assessment. However because the analytical report of the baseline assessment has not yet been published, the evaluator could not conduct this part of the evaluation; instead, the evaluator has attempted to identify what has caused the delay in publishing the report, including looking at identifying internal and external factors that may have contributed to the failure of this activity. Third, according to the terms of reference, the evaluator should identify the capacity of staff at the FONGTIL secretariat in providing sustainable organisational capacity development assistance to its members.

During the process of evaluation, the evaluator gathered that from beginning of the consultation process the design process for the questionnaire was not optimal. This information was identified through interviews with the members and some of the FONGTIL staff. According to the analytical report of the baseline assessment, there was an interviews training session conducted between 29 - 31 of January 2013; and the time provided for doing the  interviews should have been from 4th of February until 30th of June 2013;  however, this evaluator did not find any evidence that the MOCA questionnaire form had been completed by the interviewees, and during the interviews there was also no evidence that any of the interviewees had been questioned by the  interviewers. The pilot project that was conducted in Dili from 9th of October until 30th of November 2012 was not optimised to identify weaknesses of the questionnaire, and even some of the board members did not understand the methodology of the assessment. 
​
According to the interim narrative report which was published on 30 September 2013, the seventeen essential activities are correlated to each other, therefore if there is failure to implement one activity, this will affect the outcome for the other objectives.
The below table shows the findings of the evaluation.    

Internal Findings
Internal findings includes evidence that has been identified from internal activity which has been planned but not executed. All evidence has to be justified against the result of planned activities.
The main objective that is aimed for from this MOCA assessment will not be accomplished, because the six main activities that would justify the capacity of the Secretariat and its members was removed from the plan. It is therefore difficult for the evaluator to analyse the capacity development of each of the actors who play an important role. In order to bring about an outcome, this report will analyse each activity against the progress result.

Activity - M01
Beginning with the design of the IT-based database for internal use and for baseline analysis - this activity has not yet begun; according to FONGTIL the time was limited in order to implement this activity. The intention of this particular activity was not only for internal use but also for baseline analysis, which means that this IT-based database was also intentionally designed for analysis of the data from the MOCA assessment questionnaire.  However, because the independent consultant (INSIGHT) has declared that the database design was technically incorrect, no one has wanted to continue the job that was left.

Comment
The FONGTIL database is still very basic and not comprehensive; if a donor wanted to find out information about a specific NGO based in a specific region, with all the details about organisational performance, it would be difficult to find the answer from the database.

According to the interview with the Information and Technology (IT) Officer of FONGTIL, it appears that the job description for the IT Officer is to increase the knowledge and capacity of FONGTIL members about LINUX software, focusing on how to introduce the new program and not about how to fix the database system. The IT Officer has produced a manual for the new software LINUX program, which is unique as it is written in Tetum, however is this really what is expected of the IT Officer as per the job description.

Activity - S03
Regarding the Intensive training on Organisational Development for DLO’s and the Capacity Building Team, this activity was implemented, according to the FONGTIL staff, and was also reported in the interim narrative report of 2013. The idea of this activity was to improve staff capacity in the Secretariat as well as the DLO’s. The training was attended by twelve participants, including five DLO’s from Baucau, Maubisse, Liquica, Lospalos and Manufahi and seven members of MOCA’s team, as well as two staff from different divisions of FONGTIL. The training was conducted by a local private firm (INVEST PEOPLE).

Comment
After reading the report of the training, the evaluator has concluded that the training appears to have been very general, and has little engagement to the reality of FONGTIL activities. The training materials can be considered standard for any NGO staff around the world, which has been translated into Tetum. The evaluator questions the capacity of local Timorese staff to absorb the material and understand the topics. The evaluator also did not see any feedback assessment or evaluation of the methodology of the training. During the evaluation, the evaluator spent much time with the MOCA team, as well as the DLO’s from Maubisse, Baucau and Liquica. Upon attempting to explore how far they have absorbed the material from the training, there was little evidence to show that they had.

Recommendation
For the future, for this activity FONGTIL needs to appoint someone to write a detailed special report about the training, for which the content would include the objective of the training, a copy of the training materials, an evaluation of the training material and the facilitator, and details of follow up after the training. This special report could be developed into a handbook, which could become a guideline for planning and evaluation for the next training.

Activity – S04a
Training on Finance Information System (FIS) for all staff is very important and crucial for the successful financial management of organisations, therefore FONGTIL has seriously taken this activity into consideration. The activity was executed in mid-December 2013, contrary to the plan in which it should have been implemented in November. However, it was attended by 28 participants, most of whom were internal staff at the Secretariat and DLO’s of FONGTIL. The training was facilitated by the finance officer and the team, and objective of the training was to increase the knowledge of rules of financial management of an organisation, which was one of the recommendations of the external financial auditor.

Comment
The training was attended by almost all FONGTIL staff including DLO’s, which was very good. The evaluator did not see training materials and did not have a chance to speak to the Finance Officer who was busy with the financial audit at the time.

Recommendation
For future training it is recommended that two different materials are used - the FONGTIL finance manual and the finance manual from another organisation, so that the two can be compared. As mentioned in the previous recommendation, a special report should be written about the training which can be developed into a handbook of guidelines.
Activity - S06
Intensive training on Monitoring & Evaluation for all Program Officers and District Liaison Officers - this training was conducted twice with different participants, in September and October 2013. The training involved DLO’s and NGO members who are involved in monitoring networking organisations (Rede Monitorizasaun) and the facilitator for the trainings was the same person from INVEST PEOPLE.

Comment
FONGTIL already have their own guidelines for planning, monitoring, evaluation and reporting, prepared by the Logistics Manager with the President of the FONGTIL Board and approved by the Executive Director in May 2013. The evaluator has analysed the existing guidelines, which appear to be very general, and which would need more input and contribution to make them simpler and easier to understand and implement. This training could have been used to discuss and complete the existing guidelines rather than hire a facilitator to produce a new template.
According to the verification form that the evaluator received from participants during the evaluation process, fifty percent (50% of 45) respondents stated that they do have guidelines for planning, monitoring, evaluation and reporting (PMER). According to baseline report, six months after the training, fourteen percent (14% of 267) members stated that they do have PMER documents. There is not much difference between the two statistics: six months ago 38 of 267 members have said they do have these guidelines, and now 22 of 45 members state that they have them. To clarify, the 45 members are those that were involved during the evaluation process.
The evaluator recognises that is not easy to measure the outcome of this activity in the short term, because it is dependent on the ability of each participant to understand the topic, and also on organisational capability to implement the guidelines. The evaluator believes that as most of the members are in receipt of donor funding, they must be able to apply PMER principles and activities into their funding proposals.

Recommendation
In the future, FONGTIL should be more diligent in measuring training outcomes. If the DLO’s have already received the training, it should be followed up by challenging them to provide similar training in their own districts, by using similar guidelines. If the DLO from Covalima is not confident enough to facilitate the training, the methodology of an individual facilitator can be changed to a team facilitator, making it easier to provide good quality training and produce good quality guidelines.

Interim Financial and Narrative Report
The Interim Financial & Narrative Report was published on 30 of September 2013. The document clearly explains the progress of each activity; some of the explanations are very detailed. However the report is in English only, making it difficult for most of the staff to understand.

Recommendation
In future the document needs to be prepared in both languages, English and Tetum. If this is difficult to organise, the Performance Development Review document needs to be reviewed to include multi-language as an indicator for the PDR evaluation.  

Activity - S07a
Hiring of a Technical Writer for the Baseline Report - according to the 2013 interim report, the recruitment process began in March, but due to the low budget to pay for the consultant, nobody applied. After altering the budget line, finally FONGTIL awarded the job an independent consultant from INSIGHT. The contract began on 9 September and finished on 16 December.
For the process of data analysis, the independent consultant for technical reasons refused to use the SPSS program; he proposed to FONGTIL to increase payment with the reason that he would need time to re-enter the data and change the previous SPSS program into ODK. This therefore affected the due date for submission of the draft document, which was over by one month. The draft document was received on 17 of January 2014, and a few weeks after that the final document was received on 9 February. This process was very complicated, because at the beginning of the data entry process, two FONGTIL staff members received training from an SPSS specialist and, according to them, they and the specialist had though that the data entered was correct until the INSIGHT consultant said otherwise.

Comment
The evaluator has analysed this case and has concluded that from the start, the design of the MOCA project was not optimum. According to one of the former board members interviewed, it had not yet been decided how FONGTIL were going to analysis the data after the collection of the questionnaires. The SPSS program was introduced by one of the international advisors who was working with one of the international NGOs in Dili, however when the data was ready for entry into the SPSS system, the advisor who introduced the system had to leave because her contract had finished. The new advisor was not familiar with the SPSS program.

Recommendation
In future, if FONGTIL were to design a similar project, a simple program should be used for data analysis, and FONGTIL should ensure that they have enough human resources to execute the activity. The time that it takes for data entry should also be considered, and it should also be ensured that there is a budget line allocated in the proposal to hire an advisor to work together with the team from the start until the completion of the project.  
The following six activities were requested to be removed by FONGTIL, as described in the interim narrative report 2013. Even though this was the case, it is still possible to implement them if there is any funding available in the future.  

Activities – S08 & M07
Hiring of technical writers and development of three top priority capacity development modules, activity - S08, and activity - M07. These activities were requested to be removed from the 2013 Activity Plan.

Recommendation
These tasks be joined into one activity, in order to avoid confusion for the writers and for FONGTIL itself. The Terms of Reference for the technical writer did not clearly express that the consultant would need to identify three priorities of capacity development needs, however the identification of these three priorities should be not be difficult to do. Now FONGTIL has to do this process on their own, by going through again the 267 MOCA questionnaires and identifying those priorities.
According to key documents that were collected by the evaluator during the consultation process, 45 respondents of the 69 participants who attended the consultation meeting in Viqueque, Baucau, Liquica, Bobonaro and Covalima stated their top three priorities. One hundred  percent of the organisations have their own Constitution otherwise they could not be registered with FONGTIL; eighty two percent (82% of 45) stated that they have internal regulations; thirty four percent (34% of 45) have guidelines for human resources management, and forty five percent (45% of 45) have a regulation for managing their assets, while fifty five percent (55% of 45) have a finance manual and fifty percent (50% of 45) stated that they do have guidelines for planning, monitoring, evaluation and reporting.

These key documents from 45 organisations identified that the members’ training priorities are, in order of priority, training in drafting of guidelines for human resources management; drafting of guidelines for assets management; and drafting of guidelines for planning, monitoring, evaluation and reporting.

Activities M08 & M10
Intensive Training of Capacity Development Planning Facilitators – M08 and Capacity Development Planning Workshop – M10 - these two activities were requested for removal from the 2013 Activities Plan.

Recommendations
These two activities need to be combined into one activity, because to plan a workshop for capacity development of members depends upon capacity of facilitators that have received the intensive training. The capacity of facilitators is important in order to ensure that there is consistency of results.    

Activity M10a & M12
FONGTIL and 180 members (60% of 375) undertook Capacity Development Planning – M10a and Training on the Top Three Capacity Priorities for 54 members (30% of 180) – M12. These activities have been requested for removal in the 2013 Activities Plan.

Recommendations
Even though these activities were requested for removal from the 2013 Activity Plan, the evaluator considers that this could be included in the 2014 activity plan, because the final outcome of the MOCA projects depends upon these activities. To measure the capacity development of its members including staff of the secretariat depends upon this kind of activity; therefore the evaluator recommends that this activity be implemented.

Activity M13
The Advocacy Campaign was to be based upon the Baseline Results (see SO7b), however this activity has not yet been implemented because the final baseline report was delayed and has affected this activity. This activity includes producing 1000 copies of a CD Rom, and pamphlets, and can be considered to be a very big activity requiring a lot of human resources to execute. Because it has been requested for removal from the 2013 Activity Plan, it is difficult for the evaluator to assess point number two of the Term of Reference.  
​
Capacity Development Training
There are a number of trainings that were conducted by FONGTIL with the members, and some of these activities have already been identified in the MOCA project; other activities were not included as part of the MOCA project, however the evaluator still considers that the benefit of the training can be justified as the capacity development of members, measured against objectives number 1 and 3. The table below shows us that the number of participants was very small. Three trainings which were part of the MOCA project activity plan such as for Financial Information System (FIS- S04a), Monitoring & Evaluation (ME-S06), and also Organisational Development (OD-S03), were implemented in 2013. The six trainings listed below have so far capacitated 94 people, representing 54 organisations and 40 staff from the Secretariat and DLO’s of FONGTIL. Appendix 2 shows the list of training participants. 54 organisations of 267 members, is only 20% of training over a period of twelve months. Moreover this data only shows us the rate of participation itself and does not tell us anything about the level at which the participants understood the training material.

Even though the training involved 40 FONGTIL staff members, this does not mean that all FONGTIL staff have participated in the training. Table 2 shows us that of the ten DLO’s only five have attended two trainings, and another five participated in one training. Of the management team, one person attended three trainings, two attended two trainings, and another two participated in one training. In the Secretariat there was one staff member who attended 3 trainings and five staff attended two trainings, and the remaining nineteen attended one training.

Table 2 shows that the capacity development activities of FONGTIL staff is minimal - how can the Secretariat capacitate 267 members if the Secretariat themselves do not have enough capacity themselves.

External findings
External findings are external factors that have contributed to the successes and failures of the MOCA process. For example, concern about services that were supposed to have been provided by FONGTIL to their members; concern about funding for civil society from the Prime Minister’s Office; concern about the separation of powers between the Board and the Executive, etc. The evaluator aimed to identify other influences that may have caused some activities to not be implemented.  
During the process of evaluation, the evaluator also asked questions to participants regarding how far they understood the MOCA process. According to sixty seven organisations from the six districts who participated in the consultation process, forty percent (40% of 67) understood the MOCA assessment, and eleven percent (11% of 67) stated that they did not understand the objective; a large forty seven percent (47% of 67) answered that they did understand the MOCA assessment, but when the evaluator asked them to explain it, it was clear that they actually did not understand.

Dili
The interviews in Dili began on 13, 14, and 15 of February 2014. Consultation was divided into three days of face to face interviews; for each day the evaluator targeted five organisations - fifteen all together at the end of day three. The evaluator has found very variable findings, some of which are very interesting to explore. For example; seven of the fifteen NGOs interviewed in Dili stated that FONGTIL was not pro-active in introducing members to donors, and eight organisations stated that they do not need help from FONGTIL. One of the members had a creative idea which involved FONGTIL facilitating a joint proposal, combining four or five members together to apply to one donor. Two vulnerable organisations stated that they need assistance from FONGTIL to do advocacy with them in order to campaign about their rights of living. In contrast, thirteen organisations in Dili said that they do not really care if the advocacy division of FONGTIL could assist them. Two organisations in Dili had appreciated work that had been done by the DLO’s of Maubisse and Baucau, who according to them are very pro-active. Eight organisations stated that they have some experience with assessments from FONGTIL, but they did not see any results or follow up. Eight members declared that they were willing to collaborate with FONGTIL in order to facilitate capacity development training to members in the districts but they did not see any plan or engagement from FONGTIL. All fifteen members in Dili questioned their member fee payment – they believed that expenditure of the member fee should be more transparent. Only one organisation recognised that some of the FONGTIL Secretariat staff had helped them in writing a funding proposal. Thirteen organisations stated that they never receive any training from FONGTIL, or facilitated by the FONGTIL Secretariat. Two of the fifteen informed that they had previously participated in FONGTIL training, about preparing a draft manual for finance and advocacy using social networking media.

Viqueque
On 17 of February 2014, the evaluator held a consultation meeting with FONGTIL members in Viqueque district, involving eleven local organisations. The meeting used a method called focus group consultation, in which all the members including the evaluator talked and discussed on previously determined topics for discussion, which included a verification of the MOCA assessment, about the key documents as referred to in Section II; and the members’ opinion of FONGTIL services to members. Another topic was raised by the focus group participants, about access to the civil society fund. There was concern from many members about the predominance of donor funding in Dili only; how they rarely or never receive any training from FONGTIL about how to write a good proposal. One member recognised that FONGTIL had facilitated her organisation to attend an international conference overseas. Ten organisations agreed that the DLO of Viqueque is very active, however some questioned his job description. However, all of them agreed that the DLO should have his/her own office. The eleven organisations also agreed that FONGTIL should work with international NGOs in collaborating with local NGOs in implementing their programs and activities in Viqueque. They provided an example were collaboration did not occur, even though the international and local NGO have the same program and activity. Nine organisations questioned why last year only two local organisations were able to access the civil society fund from the Prime Minister’s Office. All eleven members of FONGTIL agreed that the policy around the civil society fund should be changed and adopt a flat rate or equal distribution to all NGO members with total budget amount of $25,000 per NGO.    

Baucau
The meeting in Baucau was held on 18 of February 2014, and according to the list of participants, the ten organisations who were involved in the MOCA project all participated again in this consultation meeting. Three of the ten members has criticised a lot about FONGTIL services to members, however most of these criticisms were without evidence and fact. Five organisations did not agree about the flat rate policy for the civil society fund, which had also been referred to by the members from Viqueque. One organisation supported by another four organisations agreed with the policy of equal $25,000 distribution for all NGOs to use for planned programs and activities.  

Liquica
On 19 of February 2014, a consultation meeting was held in Liquica, attended by twelve organisations. All members agreed that there is always a regular meeting every month between members and the DLO, however the topic of discussion is always around the monitoring of infrastructure projects in Liquica district; there is little discussion about how to increase member’s needs on capacity development. Eleven of twelve members stated that FONGTIL never facilitates trainings about how to draft key documents. Two members recognised that DLO does not have a clear task, because many times national and international NGOs in Dili have utilised the DLO for their own programs. Twelve members agreed that FONGTIL should facilitate meetings between donors and members in the districts;, the twelve members were also concerned that the DLO should be advocating for the member’s needs.  One member has declared that they do not get any benefits to becoming a member of FONGTIL, even though they have paid a membership fee. Three members asked FONGTIL to finance monitoring network activities (Rede Monitorizasaun). One member of Rede Monitorizasaun asked the advocacy division of FONGTIL to advocate for any problems with infrastructures projects that were identified by the group (Rede), so that there would not be any negative impact. All of members of the meeting agreed that policy of civil society fund should be that it is available equally to everyone.

Bobonaro
This consultation was attended by five local organisations on Thursday 20 of February 2014. Only thirty three percent (33%) of those members who had been involved in the MOCA assessment took part. Only five organisations attended the meeting; while fifteen were expected, the bad weather was an issue on the day. The members complained that previously they had regular meetings with the former Executive Director of FONGTIL, however this was no longer happening. Only one organisation stated that they had an effective finance manual. Two members complained that FONGTIL applied for the same funding that they did, and that FONGTIL received the money but they did not. Two members complained that they had experienced many assessments from a range of organisations, but never saw any results or follow up. Five members remembered that an NGO had collected database information; while they did not know what for, they did see the FONGTIL logo at the top of the questionnaire form, which is why they were willing to provide information for the questionnaire. Two members questioned the job description of the DLO, who they felt did not have a specific set of tasks assigned. One member was concerned that the DLO was not active in advocacy. However they all agreed that it was important that the DLO should have his/her own office. Five members agreed that the policy of the civil society fund should be equal for every member, at $25,000.
​
Covalima
This consultation was held in Suai on Friday 21 of February 2014; there were fourteen participants. Three members complained that FONGTIL was no longer pro-active in meeting with its members in Covalima. Three participants recognised that they had been involved in some training provided by FONGTIL, for example training on finance manuals, and as result one organisation already had their own finance manual and another two were in the process of drafting theirs. Twelve members recognised that the DLO was very pro-active in advocacy, especially in response to an emergency situation or crisis, and that the DLO organised a regular meeting with them and the local authority. Despite this, five organisations complained that the DLO did not follow up their proposal that was submitted to donors. Three members said, and another five members agreed, that their organisations had not received any donor funding for the past three years. Eight out of fourteen members believed that FONGTIL does not have any fixed plan for capacity development of its members. Ten NGOs had experienced the database collection, however they did not know what it was for. Thirteen NGOs agreed to propose that FONGTIL should organise a mini Annual General Meeting in Covalima district, which would involve local authorities, donors and local members. Lastly, all of the fourteen NGOs had agreed to discuss the policy of the civil society fund.
The above findings have been classified into six different categories below:

Twenty percent (20% of 67) members commented that they need help from the FONGTIL Secretariat in facilitating donor access to them. Eight percent (8% of 67) said that they do not care, that FONGTIL is not relevant to them. Twenty three percent (23% of 67) members gave negative comments about FONGTIL. Three percent (3% of 67) members expressed that they willing to helps FONGTIL to capacitate other members. Thirteen percent (13% of 67) members stated that they are appreciative of FONGTIL’s assistance to their organisations, for example through trainings and facilitations. Thirty three percent (33% of 67) members recommended many points for FONGTIL’s follow up.  

District Liaison Officer (DLO)  
During the consultation process in the six districts, the evaluator also interviewed the DLO’s from each district. All of the DLO’s agreed that the FONGTIL Secretariat in Dili should have a plan for regular trainings for capacity development for its members. Four of them complained about difficulties working without an office. Six DLO’s questioned the coordination link from the Secretariat in Dili, because they felt that at times the Secretariat forced work on specific issues on them without enough coordination with the DLO’s. Six DLO’s were concerned about the minimum amount of financial support for monitoring of activities. One DLO in particular was upset with about the lack of financial support on a specific issue, and the same DLO also complained that the Secretariat did not want to support local NGOs for fundraising. All of them believed that there was no or limited planning agenda from the Secretariat, and that most of the time there were emergency agendas or activities put in place, which sometimes interfered with other activities already planned in the districts.    

FONGTIL Board  
The consultation process involved two active and two former board members; former board members were consulted because they had been involved in and knew the process of the MOCA assessment. One of the board members had been directly involved in designing the questionnaire for the MOCA assessment; another three had known about it but had not been directly involved. Some of the board members questioned the structure of the FONGTIL Executive which they believed to be too big and inefficient, and two board members stated that FONTIL’s operational costs were too big in comparison with the budget for programs and activities. Two board members felt that the Executive did not pay enough attention to prioritising the main program, preferring to execute small grants rather than big project. Some of the board members expressed their opinion about a lack of qualification and skills in the Secretariat, meaning that it was difficult to expect them to be able to provide capacity building to other members. One member stated that the FONGTIL Constitution did not really mandate for regular meetings, therefore many times there was no regular board meetingor the meetings did not achieve a quorum.

Comment
Board candidates must have knowledge of the separation of powers between the board and the staff as executive, and the difference between their functions. Some FONGTIL staff felt that there had been some interference from the Board to the staff Executive, an example being when the board decided to terminate the contracts of FONGTIL staff members and DLO’s; this has affected some project activities, such as the publishing of the final baselines assessment report results. This is one example where lack of knowledge for interpretation of the mandates has resulted in a negative impact to other projects. 

Recommendation
The Board should consider the issue of separation of powers between the Executive and the Board at the next Annual General Meeting (AGM). In order to minimise the conflict of interest between the Board and the Executive, FONGTIL needs to re-evaluate the composition of its Board, and perhaps trial new board members from outside of the FONGTIL membership, who can provide good advice to the FONGTIL Executive, for example three independent members or experts, and four who still represent the members. This new system could be trialled for one board term.
The Board also needs to evaluate the duration of the contract for the FONGTIL Executive Director, whether three years is sufficient to measure the progress of results by the Executive Director. The evaluator would recommend a five year term.

Recommendations from this evaluation to the Management Team
In the future if there is the opportunity to implement a similar project that similar, it is recommended that FONGTIL discuss and analyse the benefits and impact to FONGTIL, before applying for the project. Before submitting the proposal, FONGTIL would need to measure the capacity of staff and available human resources, make sure that there will be enough time to implementing the project, and that there will not be, or only minimal clashes with other projects that will implemented at the same time. If lack of human resources is an issue, it would be better to recruit an advisor to accompany the project from the beginning until the end, ensuring that the cost for the advisor is included in the proposal. If the project is considered to be a big project that is likely to continue on for some time, it would be necessary to consolidate human resources and prioritise the project, while re-allocating the delivery of smaller projects to the DLO’s or other members of FONGTIL to execute.    

The structure of the FONGTIL executive needs to change, as currently it is unclear who is in charge if the director is not available. There is not a clear management team; currently each division has their own mandate, which makes it very difficult for each team to listen to the other. For example, the MOCA project involved seven FONGTIL members, including 1) Manager of Membership Service and Training (MST), 2) Office Manager, 3) Membership Services Officer, 4) Human Resources Officer, 5) Capacity Building Officer, 6) Resource Centre Officer, and 7) Manager of Finance. If we consider that the Manager of MST is to lead the MOCA team, then the Office Manager should not be in that team, because if we consider the Office Manager to be second in charge after the Director, he/she should be acting as the Program Manager who is managing all the other divisions, and also ensuring that the division managers should be reporting to and are accountable to him or her. The Office Manager can take on the role of Acting Director if the Director is not available, not the President of the Board – this is confusing for the staff and managers because decisions are coming from the President of the Board which can prejudice the competence of the Executive.    

Recommendations to the MOCA team
The Team Coordinator must be able to ensure that all activities are executed according to the established plan and time line. The Coordinator has to be responsible for the timetable, and each activity needs to be evaluated regularly, at minimum once per week. If there is an obstacle, this should be tried to be solved within the team first before taking to the office manager or director level. All team members have to have a specific responsibility for an activity and all team members should be expected to submit their plan of how to guarantee that each activity will be implemented according to the timeline.

Recommendation to the Executive Director
The Executive Director of FONGTIL needs to clarify the mandate, competence and duty and responsibility of the Executive Director as per the FONGTIL Constitution. The evaluator recommends that there is a submission put to the next Annual General Meeting for ratification of mandates and the competence of the Executive Director. The Executive Director should also be organising regular meetings with all members in the districts, and creating a healthy work environment at the FONGTIL Secretariat as well as out in the districts. If there is tension arising or problems between staff and staff, or staff with the board or staff with FONGTIL members, it is up to the Director to initiate or facilitate any activity that can re-open the lines of communication between them. 

Conclusion
The data from Appendix 2 showed that in the past twelve months, FONGTIL has organised six trainings for building the capacity of their staff and members. Three of the six trainings were included in the 2013 Activity Plan 2013, which is Organisational Development, Financial Information System, and Monitoring and Evaluation training, while the other three trainings were not a part of the MOCA project activity. A total of 94 participants attended the training, comprised of 54 participants from the membership and 40 from the Secretariat and DLO’s of FONGTIL. 54 of 267 members is only twenty percent (20%) of the FONGTIL membership that has benefitted from these trainings; twenty percent over twelve months is very low as an indicator for capacity development of its members. This indicator moreover is only for the level of participation, not yet for the level of comprehension of the material and the ability to put it into practice.

Objective number two cannot be assessed because the baseline result has not yet been published. However, the evaluator did try to ascertain the factors that had caused this delay. According to the evidence and analysis, the evaluator concluded that the project design was not optimal from the beginning, the lack of experience of FONGTIL staff also contributed to the problem, and the Board was not proactive in assisting the execution of some of the required activities. The result is that the outcome now cannot be measured.  
 
The capacity FONGTIL staff including the Secretariat and the DLO’s is minimal; according to Appendix 2 below and Table 2 above, over a twelve month period only two staff members attended training three times; twelve staff attended twice and twenty seven staff attended training once. It can be questioned how the FONGTIL Secretariat is to provide capacity development to 267 members if the staff themselves have received very minimal training.  

Above all, the removal of six main activities from the activity plan for the MOCA project has affected the ability to measure the capacity development of FONGTIL’s members. The internal factors which included the initial process for design of the project, regarding issues of time management and appropriate human resources, and the lack of experience of the staff, as well as lack of supervision by the management team has caused the failure of the project.

External factors that have contributed to the failure of the project is the complexity of problems that FONGTIL’s members face in their own organisations, and their relationship with FONGTIL, which has accumulated and worsened over time. The different situations and opinion of the 67 interviewed members can be seen in the pie chart above, most notably their ambivalent relationship with and criticisms of FONGTIL. The above recommendations regarding these external factors would go some way to repairing these relationships.
​
References
FONGTIL, 2011, Grant Contract External Actions Of The European Union, DCI-NSAPVD/2011/277
FONGTIL., 2011. ‘Grant Contract External Actions of the European Union’: DCI-NSAPVD/2011/277-972
FONGTIL., 2012. ‘Interim Narrative Report: Annex VI’.
FONGTIL.,2012. ‘MATADALAN LIBRE OFFICE’.
FONGTIL.,2012. ‘Matadalan Planeamentu, Monitorizasaun, Evaluasaun no Reportazem’.
FONGTIL.,2012. ‘Matadalan Oinsa Uza MOCA’.
FONGTIL.,2012. ‘MOCA Questionary Tools’
FONGTIL., 2012. ‘Relatoriu Annual Forum ONG Timor-Leste’.
FONGTIL., 2013. ‘Interim Narrative Report: Annex VI’.
FONGTIL., 2013. ‘Mata Dalan Papel no Funsaun Konsellu FONGTIL’, Konsellu FONGTIL.
FONGTIL., 2013. ’Matadalan Membership Membru FONGTIL’.
Soares, E., 2013. ‘FONGTIL Members Organisational Capacity Assessment: MOCA, Analytical Report’. INSIGHT.
Notaras, M., 2013. ‘Mid-term External Evaluation – 2012/2013 EU Project including FONGTIL MOCA Assessment.