Skip to main content

Part 3: Onboard BetaCorp Structured Sources

Not everything in BetaCorp is locked in unstructured text. Its Dynamics 365 and SAP systems hold clean, structured customer and billing records that can be onboarded directly — no agent required. In this part you use Integration to build the profiles, join, map, connection, and process that normalize those structured feeds into the CustomerGolden360 shape and publish them into the DataHub BETACORP source. Because these components form one cohesive integration, you build the whole thing with a single Companion prompt. The two feeds are the betacorp-customers-dynamics.csv and betacorp-billing-sap.csv files from the sample-records bundle (see Prerequisites).

What is excluded here

The ambiguous "Starke Ind." row is present in the Dynamics export, but it is intentionally dropped during normalization — it has no matching SAP billing record to join and its identity is unresolved, so it never reaches the published structured feed. It is the ambiguous identity the agent track handles through the unstructured path in Parts 4 and 6 — not Integration. The five records that publish are Northwind, Globex, Initech, Hooli, and Wonka.

Shared environment — build and test, but let the instructor publish

Publishing structured records writes into the shared BETACORP source. In the shared workshop account you build the integration in this part and verify your work in Test mode — but you do not deploy or run the live publish. Your instructor publishes the structured feed once, and you observe the result in DataHub. Do not run initial-load on the shared source. Because everyone in the room builds the same components, the prompt tells Companion to prefix every component name with your initials — replace [builderInitials] with yours. To run the publish yourself end-to-end, use your own trial (see the appendix).

Find your Integration Folder ID​

Trial Accounts

Companion will automatically create directories in the platform for the assets it creates. If you are the only user of your Trial account you can still do this section if you want to organize your assets, or you can skip past it.

To avoid crossing over and messing with other participants work, we need to tell Boomi Compaion what directory to build in. We can do this with a simple Companion prompt. You will need a couple of things.

  1. Your Lab ID (typically includes your company name)

  2. The prefix of your username (i.e. test.user@boomi.com -> test.user)

Ask Companion to find your folder given these two pieces of information. As an example:

Find my folder ID for the Boomi Lab in Integration. Use my username to locate the proper subfolder. Once you locate it, add it to our .env file.

Update our Integration Plan​

Now that we've successfully implemented (or reviewed) the existing assets in Data Hub, we need to update our plan for Integration accordingly.

  1. Run the following prompt, or a variation of it:

    Using Boomi Companion and the information we've gleaned through our work in Data Hub, update our integration plan to align.

Build the Integration​

Rather than building each component by hand, or instructing Companion for each one, we can hand Companion the plan we build in Part 1.

  1. In Companion, build the complete integration:

    Based on the plan we developed and saved, use Boomi Companion to build the integration necessary for our data unification. Follow the plan exactly, and be thorough in your work. When you're done, make sure the integration canvas is neatly organized so I can visually review. Put all assets in the folder we just identified. Do this in a subagent so we don't block the main thread.

  2. Confirm Companion built the full set: the Dynamics and SAP flat-file profiles, the per-record and batch DataHub XML profiles, the cross-reference table, the normalization map, the REST connection and POST operation, and the process wired Start → Message → Map → Decision (drop no-match) → Combine Documents → Set src → Notify → REST POST → Notify → Stop.

    Long Running Build

    The plan we developed and updated is comprehensive. Your agent will likely break this down into multiple steps, and execute it piece by piece over several minutes. This is a great opportunity to move on to Part 4 which does not utilize Companion.

    The Starke drop is a Decision route, not a map trick

    A Boomi map cannot conditionally suppress one record of a repeating target based on a missing cross-reference lookup — if you try to "drop in the map," the ambiguous Starke row still emits as an empty-taxId <customergolden360> and your batch ends up with six records instead of five. The drop must happen as a Decision step after the per-record map: records whose taxId came back empty (no SAP billing match) route to a Stop, and only matched records continue into Combine Documents. This is why the map targets a per-record profile and the <batch src="BETACORP"> wrapper is assembled after the drop — the src attribute is set on the combined batch, not in the per-record map.

Test and Verify​

  1. Run the process in Test mode in the platform. Do not deploy it — in the shared account the live publish is instructor-only.

  2. Confirm the first Notify step logs a correct <batch src="BETACORP"> payload with five normalized records — Northwind, Globex, Initech, Hooli, and Wonka. This payload is the success state for this part.

    Your test's REST publish returns 401 — expected, ignore it

    Your connection carries placeholder credentials, so the REST publish step in your test execution returns HTTP 401. That is expected and safe to ignore — the correct five-record batch payload in the Notify log is what you are verifying. Only the instructor's run, with the real Hub token set, publishes live.

    Instructor-only — deploy and run the live publish

    Deploying the process to the Prod-Cloud environment and running the live publish into the shared BETACORP source is done once, by your instructor. On a brand-new source, DataHub requires the initial-load lifecycle to run before it accepts ordinary updates. If a plain publish returns HTTP 400 ("source is not yet marked as one that can send updates"), the source needs its initial load enabled, the records loaded, and the initial load finished. Once the Hub token is set and the lifecycle is run, the five structured records publish live: Northwind, Globex, Initech, and Hooli update onto their existing Acme golden records (matched on taxId, no duplicates), and Wonka is created as a net-new golden record.

What You Now Have​

  • Flat-file profiles for the Dynamics and SAP feeds and an XML target matching the DataHub batch shape.
  • A cross-reference join that brings SAP billing fields onto the Dynamics records.
  • A normalization map and a tested process that publishes the structured batch into the BETACORP source (deployed and run live once by your instructor).
Structured sources onboarded

You onboarded BetaCorp's structured Dynamics and SAP feeds into DataHub, normalized to the common Customer shape, and watched DataHub match four of them onto existing Acme golden records and create one net-new. The structured side of the consolidation is done. Next, you will put the agent's action tools under governance through API Management — then build the agent that reads the hard customer truth locked in unstructured text.