Disable CORS for working on a microservice API with localhost

• Disable CORS for development by importing "cors" and then use it to disable cors for 'development' in each microservice

  if (process.env.NODE_ENV === 'development') { log.info('disable cors'); app.use(cors()); }

  EXAMPLE w/ cors:

  EXAMPLE w/ header options:

• Chrome extension that will let you disable CORS when you want without having to shutdown

  • Simple option: or - You can click the red button to make it active when you want to disable CORS.

  • Lots of settings, but you have to create an account:

• Try different browsers or the proxy setting for CRA.

• Create your own proxy server:

Helpful reading!

Debugging: Explain settings: CORS module settings:

Domain Services

Treetracker API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Field Data API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Stakeholder API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Earnings API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Regions API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Messaging API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Map Tile Service

Source Code:

CDN Endpoint: , also tiles2, tiles3, tiles4

Mounted Endpoint:

Grower Account Query Service

Source Code:

Mounted Endpoint:

Internal Endpoint:

Web Map Query Service Consumer

Source Code:

This queue message consumer exposes no API

Bulk Pack Services

Bulk Pack Consumer

Source Code:

This queue message consumer exposes no API

Bulk Pack Processor

Source Code:

This scheduled job exposes no API

Bulk Pack Transformer V1

Source Code:

Mounted Endpoint: not mounted

Internal Endpoint:

Bulk Pack Transformer V2

Source Code:

Mounted Endpoint: not mounted

Internal Endpoint:

Legacy Services

Admin API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Web Map API

Source Code:

Mounted Endpoint:

Internal Endpoint:

Cloud Services

Airflow API

CKAN API

Migration Sequence

1. Migrate all legacy 'tree' table data into the fielddata.raw_capture table

   1. Includes creation session and device records for these captures

2. Admin panel application: split capture verification tool from capture search tools.

3. Go through the migration process for capture verification

   1. Documentation:

   2. Steps
4. Migrate all remaining approved legacy 'tree' table data into the treetracker.capture table

   1. Apply to dev

   2. Apply to test

5. Consolidation step to fix all planting organization assignments

   1. Priority between planting org per planter, per tree, and reported in field data to be determined

   2. Clean up all capture records to have valid organization assignments in the stakeholder tool.

6. Update admin panel to using treetracker API for capture search tools.

7. *Now, ready to get the capture matching tool working in deployment*

8. GIS task: Identify very small geographic area in Freetown as a pilot for capture matching, and supply this as a spatial shape file (.geojson or .shp)

9. GIS task: Identify all the capture records in the small geographic area that are the initial captures of the tree.

10. Extract production data from small geographic area and add that data to dev and test database.

11. Data task: insert treetracker.tree records for each initial capture. Now we have trees to capture match against.

    1. Perform in dev, test, and prod

12. Implement capture matching frontend feature to allow filtering trees and captures by named geographic area. We will filter for initial use by the small geographic area defined above, which should be loaded into the regions API.

13. Now we can deploy the capture matching tool to the beta infrastructure and try it out.

Notes:

1. Some legacy planter registrations reported to not have device identifier.

   1. This is inconvenient but not fixable with the v2 bulk pack format

Schema support for db-migrate

• We decided to assign search_path to each microservice user (service and migration users) in terraform to resolve this need.

Node 16 update & test

CI/CD for dev and test environment, node affinity

Fix linting errors if present, fix linting warnings

Review state of testing

Review state of open issues

Review state of open PRs

Fix security audit reports

treetracker.grower_account.wallet

• is the same as messaging.author.handle

• is the same as (public.planter.phone || public.planter.email)

messaging.author.handle

• does not necessarily have a corresponding treetracker.grower_account.wallet

• does not have a matching (public.planter.phone || public.planter.email) if it does not have a treetracker.grower_account.wallet

Capture Matching Release Project

Capture Match road map 2021: https://matthew-fullstackgis.gitbook.io/capture-matching/

Webmap query service

https://github.com/orgs/Greenstand/projects/36

‘Ground user’ and ‘individual’ are synonyms. Also ‘person’

A mobile app user registers a wallet when using the mobile phone. Wallet registrations are uploaded along with their corresponding captures. A wallet is created for the wallet registration if it does not already exist. A grower account is created for each wallet, to track the identity of the person until they can be verified. Once a person is verified, a person record is created for them in the stakeholder system. Person records can receive payments for treetracker services.

Diagram for contract tables:

https://lucid.app/lucidchart/4beb07f7-0ef0-4bf0-b88d-5548c2cda03b/edit?page=0_0#

1. Release treetracker API into all envs

2. Web map query schema & consumer into all env

   1. Release

1. We will control available authors manually at first, by populating the authors table

2. We will control access to admins using our normal process

3. For announce and survey messages, we look up associated Grower Accounts in the treetracker API, and then match those against enabled authors

1. Update to 'verified' status requested to treetracker API

2. Start transaction against treetracker schema to insert a new capture record

Unit Tests

Integration Tests

• Only access the API, do not perform database check

  • First write seeded tests for GET endpoints

  • Then write tests for create and update that use GET endpoint to validate functionality

• APIs should return created objects, and get queryable via these object Ids

• Seeding should be done using knex, and use the same seeding scenarios provided to the frontend developers

• Best Practices

  • Do specific seeding for each integration test, don't use a generic seed that isn't for the specific test

  • Objects posted to an API should be explicitly expressed within the integration test file (?)
• Running integration tests

  • Use the test-integration-working command to specific a specific test file rather than running all

  • Use .only to specify a single test

• TODO

  • Need a solution to print response errors when expect fails on an error code

Handler

• HTTP Domain

• All error and success codes

• API payload validation

• Calls a service or a model function

Service

• Orchestration between services and the domain model

  • Database session

  • External APIs

Model

• Domain logic

• Accesses repositories

• Allowances

Repository

• Accesses the database, performs CRUD operations

• One repository for each database table in RDMS

Goal: Admin panel to not read trees table directly. Build new treetracker service with API to create capture records.

Build a new service to be the source of truth of all approved captures.

Field data schem

Ingestion Process by Entity Type

wallet_registration

Wallet registrations need to be stored in the field data schema in the wallet_registrations table, but they also need to be used to upsert (create or update) grower account records in the treetracker API. A wallet registration also needs the grower account id field populated.

Suggested workflow:

1. Bulk pack transformer calls PUT on grower_account in treetracker API for grower account with identifier that matches wallet

   1. If account exists, update

      1. Only update the name, phone, and email for now

device_configuration

device configuration records are simply inserted in v2, but in v1 they were upserted.

Suggested Workflow

1. POST device configuration payload to bulk-pack-transformer-1

   1. bulk-pack-transformer-1 returns 200 if already present

2. POST device configuration payload to field data API

session

session does not exist in the v1 bulk-pack-transformer.

Suggested Workflow

1. POST to session path on field data API

   1. field data API returns 200 if already present

2. Return 200

capture

1. POST capture data to bulk-pack-transformer-1

   1. If data exists this API returns 200, else a new record is inserted

   2. This API also propagates the record to field data API, where a queue message is emitted

Orchestration Diagram

Bulk Pack Format v2 :

Improved orchestration architecture

Deprecated proposal for orchestration

Nullable Columns for Field Data Tables

Session

• track_url

• organization

Wallet Registration

• phone or email is nullable, but not both

Raw Capture

• session_id is nullable for v1 endpoints only

• reference_id

• note

Device Configuration

• No nullable fields

Workflow Notes

Insert 'tree' as 'capture' will fail until corresponding device and registration records have been inserted, because there's no session Once device and registration records are inserted, then it is possible to insert a session id using SESSION UUID = device_identifier + planter_identifier In v2 bulk pack endpoint /v1/tree is responsible for ensuring that the session record exists for a v1 tree. This session can then be used as the session for inserting the v1 'tree' record as a v2 'capture' record

1. process sends tree record to /v1/tree in v2

2. v2 calculates SESSION UUID = device_identifier + planter_identifier using uuid-string

3. v2 checks for a session that matches this

Release

Old versions:

• greenstand/bulk-pack-processor:1.2.7

• greenstand/bulk-pack-transformer:1.6.2

**Capture**

**Tree**

**Tag**

**Capture_Tag**

**Tree_Tag**

**Grower_Account**

**Status Enumeration** active deleted Session - Field Data Schema Device Configuration - Field Data Schema