Data extraction

#1

Hello,

For my project entitled as “Pilot Pre-Test” there 41 runs and I would like to extract this data. What should I do?

Thank you in advance.

#2

Hi Busra,

We are currently looking into on how to retrieve past run results.
We’ll let you know this week after we further investigate.

Regards,
Geromel

#3

Hi Busra,

Thanks for the note. Are you trying to display the runs with a standard page in the simulation?

There’s two different ways to do this.

(1) Create a new version of the sim, and use the “Run Comparison” template. This is the easiest approach. it’s set up to allow you to create and compare multiple runs. Note however that this does not allow you to go step by step through the sim.

(2) Create a custom HTML page and link it into the sim. This requires you to do some Javascript programming. You will need to use Epicenter.Js, create a run manager, retrieve the data, and build a custom table. You can find out more about our Javascript APIs here.
https://forio.com/epicenter/docs/public/api_adapters/

As an important note - data for model variables is not stored by default. Normally when you view a run in a simulation the run is replayed first, then the data is displayed.

In order to show a table of data comparing the values from different runs, you need to tell Epicenter to save the variable values in the database. You do this by creating / editing a “config” file in the model directory that marks the “saved” property for each variable you wish to retrieve.
https://forio.com/epicenter/docs/public/model_code/context/#ctx-save

Let us know more specifics about what you are trying to do and I can offer further suggestions.

WILL

#4

Hello,

Thank you for the answer, I will look in more detail to links that you send.

Basically, I want to have an Excel file (like normally I have with Vensim) with all the variables of the model, for each participant.

#5

Hello,

So I have checked the documents that you send but I am still not sure how to do this.

At the end, I want to have all the model variables for the whole simulation duration, for each participant.
How do I differenciate the participants? How can I get my data like this?

Thank you,
Busra

Data extraction is not possible using API
#6

Hi Busra,

  1. You’ll need to whitelist which variables you want to save/export from your model, by listing it in a configuration file. I see you’ve already started one here, but here’s the documentation for this just in case.

  2. There’s no built-in “export to excel” button so you’ll need to do some custom work to get this o
    ut - the simplest option is to get this from the API; for e.g. if you log into Epicenter and click on this link
    https://api.forio.com/v2/run/hec-regulation/pilot-pre-test_v2/;saved=false?include=Time,Thermal Share of Generation

it should give you a “json” list of data with the variables you need. You can add more variables after the comma, as long as those variables are also included in your configuration file. If you copy what you see in that link and paste it in a “json to csv” converter (like https://json-csv.com/) you should be able to get an excel file out.

Note: The API link above gives you all runs; if you have a lot of runs it’ll be slow and inefficient to get them all that way. If you anticipate a lot of users/runs, you’ll want to divide your users into groups and filter by group. Can you let us know more about your expected usage?

#7

Hello,

I extract my data using the methodology you described here. First I check the API and then I use https://json-csv.com/ to convert it to Excel.

However, when the file is larger than 1 MB this website asks for a payment. I would like to use this website because I prepare all my statistical analysis with its output. Therefore, I would like to delete some previous rounds to reduce the size of the file. How can I do it? Or what else I can do to reduce it?

Thank you

#8

Hi Busra,

You can request fewer variables or fewer runs.

If it isn’t many runs, you can get data for one run at a time by putting the run ID in the URL, e.g.

https://api.forio.com/v2/run////variables?include=

#9

Hello,

I want to extract data for my project named realnov_gp. I use this link to extract it
[https://api.forio.com/v2/run/hec-regulation/realnov_gp/;saved=false?include=Time,][1]

[1]: https://api.forio.com/v2/run/hec-regulation/realnov_gp/;saved=false?include=Time, and then I write the variables that I want to extract.
But this gives me only the runs of today (200 run), however I can see that the total number of run is 464. How can I get all of them?

Thank you,
Busra

#10

Another question: where do I see run ID?

#11

Hi Busra,

The API only returns the last X runs by default, but you can request more like so

https://api.forio.com/v2/run/hec-regulation/realnov_gp/;saved=false?include=Time&startrecord=100&endrecord=200 which means give you a 100 runs from 100 to 200. The “endrecord” is optional and if you leave it off you’ll get the last 100 runs by default. You can use this to construct different urls to slice data differently.

-Naren

#12

Hello,

When I use this “&startrecord=100&endrecord=200” at the end of my URI, I have the same output when I put “&startrecord=1&endrecord=500” and “&startrecord=400&endrecord=500”.
Why startrecord is not working in my case?

#13

Hi Busra,

I haven’t exhaustively checked for dupes, but the I get different runs for the first and last items when I change the start/endrecord. See screenshot. Are you not seeing the same thing?

#14

Is it possible for you to send me all my data from 27, 28 and 29 november for my 4 projects as .csv files?

#15

Hello,

When I try to extract my data using:
https://api.forio.com/v2/run/hec-regulation/p5/;saved=false?include=Time,Thermal%20Share%20of%20Generation

All I get is that: []

Why? What should I do?

#16

Based on your email correspondence with David, please let us know if this is now resolved.
Regards,
Geromel

From: Busra Gencer
Date: Mon, Oct 29, 2018 at 2:42 PM
Subject: Re: URGENT: API Data extraction
To: David

Hello,
This solution works, thank you.