Demystifying event streams: Transforming events into tables with dbt

Contact Details
Charlie Summers in dbt slack or<EMAIL_ADDRESS>I have read the dbt Developer Blog contribution guidelines.

[X] I have read the dbt Developer Blog contribution guidelines.

Which of these best describes you?

[X] I am a dbt Community member or partner contributing to the Developer Blog
[ ] I work for dbt Labs and am creating this issue for a community or marketing approved piece.

What is the topic of your post?
This article discusses how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. In it, I’ll share:

Why you may want to use an architecture like this.
How to structure your event messages.
How to use dbt macros to make it as easy as possible to onboard new domain models.
To learn more, check out my talk at Coalesce 2022!

Targeted at people working with event streams within dbt.
Link to an initial outline.
First draft: https://docs.google.com/document/d/18TQ64LQNBVWqoKwhDNukLBA-fs4XkxbB4xb5ZfwMI8Q/edit

Charlie this is a killer post!! Thank you for putting it together. It clarifies a ton of concepts super well and really goes deep on how you solved the problem.
If there's one area I'd love to see a little bit more detail on it's how exactly the data contracts were decided on, encoded and enforced. This is a topic of huge personal interest atm and would love to hear just a bit more there. Once that is done, I think we're looking really great here and can start getting this ready to publish.

@jasnonaz just finished making the requested edits! I also added a section called "Events as Contracts" - let me know if it has the info you're interested in. Happy to elaborate more if you think it's worth it! :)

This looks fantastic! Ready from our end to get this published @reallocf. Just give me a 👍 and we can get it loaded up.

@reallocf Hey Charlie, I'll be opening a PR for this content so we can get it published. I'll need access to the document to be able to do that. Let me know when you're ready to share this doc with me!

Thanks @jasnonaz and @johnblust ! I'm getting final sign-off from my marketing team that it looks good on their end as well. Will get back with more info soon 👍

Cool, I'll hold off on creating a PR for this until we get the 🟢

@johnblust @jasnonaz I made some very small tweaks on my end, feel free to review again as a final pass. But we're all clear from Marketing! They're prepping a copy of the post for our blog - @jasnonaz you mentioned we should link them together somehow, let me know what you need from me to help get that done.
Thanks so much for all y'all's help! Really excited to get this up and posted! 🎉


Failed to establish a new connection (user-defined metadata / tags on blob not getting displayed)

Related command
a) az storage fs file metadata show -p {path} -f {filesystem} --account-name {account-name}  --auth-mode login
b) az storage blob tag list -n {path} --container-name {filesystem/container} --account-name {account-name} --auth-mode login
Describe the bug
Error message displayed:
a) <urllib3.connection.HTTPSConnection object at 0x7fbdd5228190>: Failed to establish a new connection: [Errno -2] Name or service not known
b) <urllib3.connection.HTTPSConnection object at 0x7f351d03d7e0>: Failed to establish a new connection: [Errno -2] Name or service not known
To Reproduce
Login to the VM with the customised role (see permission granted below): az login --identity
Run either of the commands above (a) or (b)
Expected behavior
The command should display the key and values for
a) Metadata
b) Blob index tags
Environment summary
{
"azure-cli": "2.39.0",
"azure-cli-core": "2.39.0",
"azure-cli-telemetry": "1.0.6",
"extensions": {
"storage-blob-preview": "0.6.1"
}
}
Additional context
Permissions:
Microsoft.Storage/storageAccounts/fileServices/read
Microsoft.Storage/storageAccounts/blobServices/containers/read
Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read
Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/read

@evelyn-ys for awareness

az storage blob tag list command works for common storage account not ADLS Gen2 account.
But az storage fs file metadata show should work for ADLS Gen2 account:

Can you confirm that the filesystem and path exist?

@evelyn-ys,
Thank you for your reply.
Storage account is not ADLS Gen2.
Yes, the path exist as I can use other commands (e.g. az storage blob list).
Regards,

@montanoeuse Thanks for confirming that you are using non-ADLS gen 2 storage account. I tried to create a repro using Standard NonHNS storage account and I was able to get the required information:
I am sharing the command here if that helps:
az storage blob metadata show --container-name MyContainerName --name MyBlobName.txt
{
"test": "myvalue"
}
az storage blob tag list -n new.txt --container-name MyContainerName --account-name MyAccountName
{
"IsLatest": "true",
"test": "one"
}
az storage blob filter --account-name MyAccountName --tag-filter ""@container"='MyContainerName' and "IsLatest"='true' and "test"='one'"
[
{
"containerName": "MyContainerName",
"name": "MyBlobName.txt",
"tags": {
"IsLatest": "true",
"test": "one"
}
}
]
Are there any proxy / firewall in your environment ? If yes, could you please try it from a different network ?
Could you please run the same above command with the --debug switch and share the complete output here ?

@montanoeuse The error seems to be specific to establishing the network connection. Also since you confirmed that the same commands work fine on another environment, It seems like the issue is only related to your enviornment. I could see the below in your debug output:

urllib3.connectionpool: Starting new HTTPS connection (1): {account_name} .blob.core.windows.net:443

Could you confirm if you had entered the right account name ?
Could you please send me the complete debug output over email as-is (without modifying the parameters) ? Here is my email: navba @ Microsoft . com
Awaiting your reply.

@montanoeuse Thanks for sharing all the debug logs details over email. I am adding the Service team to assist on this further.
We have further isolated that --auth-mode key and with  AZURE_STORAGE_KEY and AZURE_STORAGE_ACCOUNT environment variables it works fine. We have also checked that the port is open and connectivity is successful. Also the nslookup is working fine.
This successful test was performed on the same environment where it fails while listing the tag.
@xgithubtriage Could you please look into tis further and provide an update on this ? Awaiting your reply.

Hi. good day! any update on this?

@xgithubtriage Please share an  update on this.


localize() should take a default value as well

In the event that the message key is not in your bundle, you might want to provide a default to use, instead of falling back to the key string itself.

Chatted with @danteoh about this awhile back. Argument was made that this should live in the bundle and not in the formatter so that it was easier to see things that were not localized vs attempted localized but missing.


sdwan command is crashing

I have been using sastre and sastre-pro tools in my Python virtual environment and it was working as expected, but now both sastre and sastre-pro are not working and it's crashing.
I tried to install sastre and sastre-pro on my new Python virtual environment and after the installation I am unable to execute any commands and getting below error :
(sdwan) [1211656@azlawxasp00001 sdwan]$ sdwan --version
Traceback (most recent call last):
  File "/home/1211656/virtualenvironment/sdwan/bin/sdwan", line 5, in
    from cisco_sdwan.main import main
  File "/home/1211656/virtualenvironment/sdwan/lib64/python3.9/site-packages/cisco_sdwan/main.py", line 22, in
    from .tasks.implementation import *
  File "/home/1211656/virtualenvironment/sdwan/lib64/python3.9/site-packages/cisco_sdwan/tasks/implementation/init.py", line 11, in
    from ._attach_detach import TaskAttach, TaskDetach, AttachVsmartArgs, AttachEdgeArgs, DetachVsmartArgs, DetachEdgeArgs
  File "/home/1211656/virtualenvironment/sdwan/lib64/python3.9/site-packages/cisco_sdwan/tasks/implementation/_attach_detach.py", line 337, in
    class AttachVsmartArgs(AttachDetachArgs):
  File "/home/1211656/virtualenvironment/sdwan/lib64/python3.9/site-packages/cisco_sdwan/tasks/implementation/_attach_detach.py", line 340, in AttachVsmartArgs
    template_filter: Callable = const(DeviceTemplateIndex.is_vsmart)
  File "/home/1211656/virtualenvironment/sdwan/lib64/python3.9/site-packages/cisco_sdwan/tasks/models.py", line 49, in const
    return Field(default_value, const=True)
  File "/home/1211656/virtualenvironment/sdwan/lib64/python3.9/site-packages/pydantic/fields.py", line 723, in Field
    raise PydanticUserError('const is removed, use Literal instead', code='removed-kwargs')
pydantic.errors.PydanticUserError: const is removed, use Literal instead
For further information visit https://errors.pydantic.dev/2.0.3/u/removed-kwargs
I have observerd that previously sastre-pro was using pydantic-1.10.7 but now it’s using pydantic-2.0.3.
Could you please help me with this.

I have investigated further and found the solution.
The root cause of the issue is :
The sastre-pro installation command "pip install --upgrade ./Sastre-Pro" is downloading pydantic-2.0.3 instead of pydantic-1.10.7
Resolution
Install required Python package using the requirement.txt file (which is already present under Sastre-Pro_1.21.2/Sastre-Pro)
"pip install -r requirements.txt"
This step is missing from the README.txt file, please add this step in the file.


show function signature in vim

how can i show the function signature in vim like the emacs. need some config in vimrc or should be a new enhance

i found it by myself!

how did you do it?

let g:racer_experimental_completer = 1 in .vimrc


Certain entities not displaying

DSC-Team4
Repo to collaborate on the team activity for the final round task of GDSC Recruitment 2023
🔗 Link to the website
❌ Problem: unable to load 2 images in the NFT-Gating section
✅ But its working properly in local host
Images :

images/pr1.png
images/pr2.png

⚒ Try to host in your repo and try to fix the bug if possible

Deployed the website through Vercel. No discrepancies discovered yet.


Automatically bind a UserControl to the parent Window

If the UserControl is bound to it's parent Window, then the WindowLocator can close the Window from the UserControl without knowing it's parent.
Call this from the UserControl to close it's parent Window:
var windowLocator = Ioc.Current.Resolve<IWindowLocator>();
windowLocator.CloseWindow(this);

Implemented in commit 86c67f5f687efeec00c6f849dd516c26072a6b2f.


New package: ACSets v0.1.0

Registering package: ACSets
Repository: https://github.com/AlgebraicJulia/ACSets.jl
Created by: @epatters
Version: v0.1.0
Commit: 672186a05a6b298417223d4c1cb01d34fd59ab92
Git reference: HEAD
Description: ACSets: Algebraic databases as in-memory data structures

Sorry for the trouble, but I would like to request that this PR be manually merged on the grounds that

The two packages concern very different domains (databases/data structures and finance)
"ACSets" is a made-up word (short for "attributed C-sets"), so in any case is unlikely to be confused with other things
We've been using this word in our software and writings for several years now

Thanks!

Using the "what are the words you use to describe it" method, it seems like AttributedCSets would be a better name.

+1 to AttributedCSets.jl

Thanks for the suggestions. However, the AlgebraicJulia team would prefer the original name since that is the name that we have been using and are now accustomed to.
Can this PR please be manually merged?

https://julialang.github.io/Pkg.jl/dev/creating-packages/#Package-naming-guidelines-1

Point taken. We'll rename the package (name TBD). This PR can be closed. Thanks.

@epatters @rafaqz actually I believe changing the name here goes against the package naming guidelines. Pointedly guideline 3: "Packages that provide most of their functionality in association with a new type should have pluralized names."
This package purely exports the ACSet type and functionality for this type. Therefore it should be named ACSets.jl
This would also align with the Python implementation package py-acsets which we would like to maintain a common naming convention with.

I agree with @mehalter that guideline 3 exceeds the value of guideline 1 in this case. AttributedCSets is still jargon, since CSet is a play on the name G-Set for a group action, which is jargon from abstract algebra.

Sure! That makes sense.
I was making the point that wanting the name anyway should be in reference to the guidelines and your situation, not just a preference. Because everyone prefers their own idea.
Now you just need to convince someone who actually has commit rights :)


USER STORY: Registration Page

As a user, I would like to be able to have a registration page that will allow me to sign up for the application.
Describe the solution you'd like
We need a basic Registration page that will have an associated form. Information on that form will be potentially:
First Name: Nick
Last Name: Di Pietro
Student ID Number:  123-45-6789
Student or Faculty:  Student
Email Address<EMAIL_ADDRESS>Make sure you take note that names will be strings, id number is only Integer, Student or Faculty can be a drop down menu with 2 selectable options, and email address can be .edu, .com, .org, or any of the normally associated endings. Be sure to use type safety to avoid any potential bugs that arise from a client inputting incorrect information.

Deleted duplicate user story #59 as this one already had a PR associated with it.


how to fix redisson specified at https://github.com/redisson/redisson/issues/4441

Hi,
I've had redisson issue in Devicehive frontend and Auth, version latest 4.1.0.
And I found the issue fixed at https://github.com/redisson/redisson/issues/4441. Please show me how to update it in frontend and Auth. Currently I use Devicehive docker image 4.1.0.
Thank you.

I follow instruction mvn clean package to build devicehive from git, also from release v4.0 zip file. But I found that boot.jar file created different from boot.jar on device docker image. And it could not run due to database schema related,
2023-09-24 04:28:52.947 [main] INFO   c.d.a.DeviceHiveBackendApplication - Starting DeviceHiveBackendApplication v4.1.0 using Java 17.0.2 on bcf01f6fcf6c with PID 12 (/opt/devicehive/devicehive-backend-4.1.0-boot.jar started by root in /opt/devicehive) 2023-09-24 04:28:52.967 [main] INFO   c.d.a.DeviceHiveBackendApplication - The following 1 profile is active: "ws-kafka-proxy-backend" 2023-09-24 04:28:53.040 [background-preinit] INFO   o.h.validator.internal.util.Version - HV000001: Hibernate Validator 6.2.5.Final 2023-09-24 04:28:56.508 [main] INFO   o.s.d.r.c.RepositoryConfigurationDelegate - Multiple Spring Data modules found, entering strict repository configuration mode 2023-09-24 04:28:56.517 [main] INFO   o.s.d.r.c.RepositoryConfigurationDelegate - Bootstrapping Spring Data Redis repositories in DEFAULT mode. 2023-09-24 04:28:56.609 [main] INFO   o.s.d.r.c.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 46 ms. Found 0 Redis repository interfaces. 2023-09-24 04:28:59.497 [main] WARN   io.undertow.websockets.jsr - UT026010: Buffer pool was not set on WebSocketDeploymentInfo, the default pool will be used 2023-09-24 04:28:59.577 [main] INFO   io.undertow.servlet - Initializing Spring embedded WebApplicationContext 2023-09-24 04:28:59.577 [main] INFO   o.s.b.w.s.c.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 6395 ms 2023-09-24 04:29:00.719 [main] INFO   o.f.c.i.license.VersionPrinter - Flyway Community Edition 8.5.13 by Redgate 2023-09-24 04:29:00.719 [main] INFO   o.f.c.i.license.VersionPrinter - See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.13 2023-09-24 04:29:00.719 [main] INFO   o.f.c.i.license.VersionPrinter -  2023-09-24 04:29:00.741 [main] INFO   com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting... 2023-09-24 04:29:01.121 [main] INFO   com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed. 2023-09-24 04:29:01.177 [main] INFO   o.f.c.i.d.base.BaseDatabaseType - Database: jdbc:postgresql://postgres:5432/postgres (PostgreSQL 14.7) 2023-09-24 04:29:01.308 [main] INFO   o.f.core.internal.command.DbValidate - Successfully validated 39 migrations (execution time 00:00.079s) 2023-09-24 04:29:01.350 [main] WARN   o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Found non-empty schema(s) "public" but no schema history table. Use baseline() or set baselineOnMigrate to true to initialize the schema history table. 2023-09-24 04:29:01.353 [main] INFO   com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated... 2023-09-24 04:29:01.362 [main] INFO   com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed. 2023-09-24 04:29:01.385 [main] INFO   o.s.b.a.l.ConditionEvaluationReportLoggingListener -

Hi,
It happens because DB already exist and we disabled autoupdate for data protection.
You can enable it by editing properties files.
Before it it's better to make backup of existing database (if you need it)
spring.flyway.baselineOnMigrate=true
in
devicehive-auth/src/main/resources/application.properties
devicehive-backend/src/main/resources/application.properties
devicehive-frontend/src/main/resources/application.properties

Yes, I did it but only for backend, and enabled spring.flyway.baseline-version=3.4.12 then it succeeded. And with redisson 3.18.0 was built successful. I faced this issue, https://github.com/redisson/redisson/issues/4451
After some days, randomly, maybe up to 1 month, then the issue occurs.
Currently I build with redisson 3.18.0, and wait for some days/weeks.
Thank you.


slade-git 3.1.2 -> 3.2.0

Motivation for this change
Brings slade-git up-to-date with the current development version; now built against gtk3.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Pinging maintainer @ertes

ci error.


Patch DDL generation doesn't detect when a duplicate field pgType changes

When changing a duplicate field's dbtype no differences are detected when generating a patch or asserting the schema configuration.
For example changing a duplicate field definition:
From .Duplicate(r => r.Artist) to .Duplicate(r => r.Artist, "varchar(15)")
I would expect an ALTER COLUMN statement to be generated, and possbly the functions and index to be regenerated?
I haven't tested with any other changes to the storage model so there may be other changes that aren't detected too.
Does this sound sensible to you? If so I'm happy to get stuck in and see if i can come up with an implementation.

@richardoliver-umusic sorry for late answer, if you'd like to provide PR for that then we'll be more than happy to take it.

This might already work, but you'd need a test to prove it.

Thanks for following up on this, I'll try and get around to testing it this week.
Apologies for not replying to you back in 2019 @oskardudycz

Hi @jeremydmiller I briefly looked into this with version 4.3.1 but it didn't seem to be recognising the dbtype change as a reason to update the schema still.
I don't have any time to look into it further or create a test case unfortunately, but it's only come up once for our team in the last 3 years so its not high priority for us. Therefore I fully understand if you want to close the issue.


Added my variant to the README

I know I'm a bit late but it just took a bit longer than planned.
I added a choice of fonts, vehicles 'split' and 'join' to accommodate for the needs of the new string and you can choose any colours you want.

Fantastic, thank you!!


Blog updates

Removed unnecessary posts
Added new content scheduled for release on mondays

The dangers of CDN's (jan 30th)
Taking your HTML elements further (TBD)
How to cheat at CSS (TBD)
References and pointers (TBD)

Updated blog styling
Added taxonomies and integrated them with blog

Authors
Tags

Added blog features

Blog tagging
Share links
RSS feed subscription
mermaid support

Added sitemap generation support
Updated documentation to include details about new features

DO NOT MERGE YET, still need:

Video styling
Documentation for videos
Structured data
Documentation for structured data


Partner review

Please review the code
Any suggestion is welcome

https://github.com/Somdotta07/To-do-List-review/blob/a544b0faea6b6785d56eb3847f10dfe5d2f57541/src/style.css#L8-L10
This css code is referenced a lot of times trough your CSS,
You c create a single class and add it to all HTML elements that need it.

https://github.com/Somdotta07/To-do-List-review/blob/a544b0faea6b6785d56eb3847f10dfe5d2f57541/src/index.js#L24
Instead of a for loop, You can use a forEach loop to loop over an array.

https://github.com/Somdotta07/To-do-List-review/blob/a544b0faea6b6785d56eb3847f10dfe5d2f57541/src/add_rem.js#L18
When redeclaring JS variables, it's better to use let

Thank you for your feedback and time :smiley:

Status: Approved :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
Hi Somdotta,
Congratulations 🎉 Thank you for implementing changes suggested previously 🙏
Happy Coding 👏👏👏


[Bug]: Stage 5 crashed with num_planets > 1

FAQ check

[X] Yes, I checked the FAQ and my question has not been addressed.

Instrument
Light curve fitting (Stages 4-6)
What happened?
I installed the latest version:
`
commit b79744612deca1dde10cad3b5e8fd666245ce606 (HEAD -> main, origin/main, origin/HEAD)
Merge: 6919a484 6e14b1ff
Author: Kevin Stevenson<EMAIL_ADDRESS>Date:   Tue Sep 24 16:48:31 2024 -0400
Merge pull request #701 from kevin218/dev/tjb3

Better documenting multi-channel and multi-planet EPF settings

`
I reran stages 2 - 4 using new .ecf files to avoid any potential problems. Everything looked good.
On Stage 5, I added  num_planets and pl1 where needed as described in the updated documentation. Stage 5 crashed.
Error traceback output
(eureka) dlong@hikaru obs3 % python run_eureka_t1be_3.py
WARNING: There are 8 metadata save files in the folder: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/
Using the metadata file inside: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7/
and will consider aperture ranges listed there. If this metadata file is not a part
of the run you intended, please provide a more precise folder for the metadata file.
Finished loading parameters from /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7//S4_t1be_3_ap3_bg7_LCData.h5
Starting Stage 5: Light Curve Fitting
Input directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7/
Output directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage5/S5_2024-09-26_t1be_3_run1/ap3_bg7/
Copying S5 control file
Copying S5 parameter control file
Starting Channel 0 of 47
=========================
Starting dynesty fit.
Traceback (most recent call last):
File "/Users/dlong/DataAnalysis/JWST/eureka/t1be/obs3/run_eureka_t1be_3.py", line 31, in
meta = s5.fitlc(eventlabel, ecf_path=ecf_path)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 473, in fitlc
meta, params = fit_channel(meta, time_temp, flux, channel,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 964, in fit_channel
lc_model.fit(model, meta, log, fitter='dynesty')
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/lightcurve.py", line 181, in fit
fit_model = self.fitter_func(self, model, meta, log, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/fitters.py", line 856, in dynestyfitter
start_lnprob = lnprob(freepars, lc, model, prior1, prior2, priortype,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 206, in lnprob
ln_like_val = ln_like(theta, lc, model, freenames)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 85, in ln_like
model_lc = model.eval()
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/Model.py", line 370, in eval
flux = component.eval(channel=channel, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/AstroModel.py", line 400, in eval
starFlux = self.transit_model.eval(channel=chan,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/BatmanModels.py", line 139, in eval
m_transit = self.transit_model(pl_params, time,
File "/Users/dlong/miniconda3/envs/eureka/lib/python3.10/site-packages/batman/transitmodel.py", line 138, in init
self.ds = _rsky._rsky(self.t_supersample, params.t0, params.per, params.a, params.incpi/180., params.ecc, params.wpi/180., self.transittype, self.nthreads)
TypeError: must be real number, not NoneType
(eureka) dlong@hikaru obs3 %
What operating system are you using?
Mac OS Sonoma 14.6.1
What version of Python are you running?
 Control File for Stage 5: Lightcurve Fitting
Stage 5 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-5
ncpu            12 # The number of CPU threads to use when running emcee or dynesty in parallel
multwhite		False  # Are you simultaneously fitting multiple white lightcurves? (if so, use inputdirlist and see docs)
allapers        False  # Run S5 on all of the apertures considered in S4? Otherwise will use newest output in the inputdir
Manual clipping in time
manual_clip     None    # A list of lists specifying the start and end integration numbers for manual removal.
fit_par         ./S5_t1be_3.epf   # What fitting epf do you want to use?
fit_method      [dynesty]                   #options are: lsq, emcee, dynesty (can list multiple types separated by commas)
run_myfuncs     [batman_tr,polynomial]      # options are: batman_tr, batman_ecl, sinusoid_pc, quasilambert_pc, poet_tr, poet_ecl, poet_pc, fleck_tr, expramp, polynomial, step, xpos, ypos, xwidth, ywidth, lorentzian, damped_osc, and GP (can list multiple types separated by commas)
compute_ltt     False                       # options are: True (correct model for the light travel time effect), or False (ignore the light travel time effect)
num_planets     2
force_positivity False                      # Optional boolean for sinusoid_pc and poet_pc models.  Set True to force positive phase variations.
Limb darkening controls
IMPORTANT: limb-darkening coefficients are not automatically fixed then, change to 'fixed' in .epf file whether they should be fixed or fitted!
use_generate_ld  None  # use the generated limb-darkening coefficients from Stage 4? Options: exotic-ld, spam, None. For exotic-ld and spam, the limb-darkening laws available are linear, quadratic, 3-parameter and 4-parameter non-linear.
ld_file          None  # Fully qualified path to the location of a limb darkening file that you want to use
ld_file_white    None  # Fully qualified path to the location of a limb darkening file that you want to use for the white-light light curve (required if ld_file is not None and any EPF parameters are set to white_free or white_fixed).
General fitter
old_fitparams   None # filename relative to topdir that points to a fitparams csv to resume where you left off (set to None to start from scratch)
lsq fitter
lsq_method      Powell  # The scipy.optimize.minimize optimization method to use
lsq_tol         1e-7    # The tolerance for the scipy.optimize.minimize optimization method
lsq_maxiter     None    # Maximum number of iterations to perform. Depending on the method each iteration may use several function evaluations. Set to None to use the default value
emcee fitter
old_chain       None    # Output folder relative to topdir that contains an old emcee chain to resume where you left off (set to None to start from scratch)
lsq_first       True    # Initialize with an initial lsq call (can help shorten burn-in, but turn off if lsq fails). Only used if old_chain is None
run_nsteps      1000
run_nwalkers    200
run_nburn       500     # How many of run_nsteps should be discarded as burn-in steps
dynesty fitter
run_nlive       'min'    # Must be > ndim * (ndim + 1) // 2
run_bound       'multi'
run_sample      'auto'
run_tol         0.1
PyMC3 NUTS fitter
exoplanet_first False   # Initialize with an initial exoplanet optimizer call (generally not recommended, but can sometimes be helpful if your initial manual guess is quite poor)
tune            3000
draws           1500
chains          3
target_accept   0.85
#GP inputs
kernel_inputs   ['time'] #options: time
kernel_class    ['Matern32'] #options: ExpSquared, Matern32, Exp, RationalQuadratic for george, Matern32 for celerite (sums of kernels possible for george separated by commas)
GP_package      'celerite' #options: george, celerite
Plotting controls
interp          False   # Should astrophysical model be interpolated (useful for uneven sampling like that from HST)
Diagnostics
isplots_S5      5       # Generate few (1), some (3), or many (5) figures (Options: 1 - 5)
nbin_plot       None    # The number of bins that should be used for figures 5104 and 5304
hide_plots      True    # If True, plots will automatically be closed rather than popping up
verbose         True    # If True, more details will be printed about steps
Project directory
topdir          /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/
Directories relative to topdir
inputdir        Stage4   # The folder containing the outputs from Eureka!'s S4 pipeline (will be overwritten if calling S4 and S5 sequentially)
inputdirlist	None     # list of extra folders to read in from after inputdir for multwhite fits. Must be strings.
outputdir       Stage5
inputdirlist    []       # A list of additional Stage 4 input folders to use in a joint white lightcurve fit (e.g., NIRSpec NRS1+NRS2, or NIRCam+MIRI). Only relevant if multwhite is set to True
Stage 5 Fit Parameters Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-5-fit-parameters
Name       Value         Free?          PriorPar1    PriorPar2    PriorType
"Free?" can be free, fixed, white_free, white_fixed, shared, or independent
PriorType can be U (Uniform), LU (Log Uniform), or N (Normal).
If U/LU, PriorPar1 and PriorPar2 represent upper and lower limits of the parameter/log(the parameter).
If N, PriorPar1 is the mean and PriorPar2 is the standard deviation of a Gaussian prior.
 See docs for details!
For fleck: can only have one spot contrast (spotcon), spotnpts is the number of temporal points to evaluate at (~200-500)
For starry: assign one spotcon per spot (spotcon, spotcon1, etc), spotnpts is the ydeg of the star map (~30)
------------------
spotstari    90       'fixed'
spotrot      100      'fixed'
spotnpts     300      'independent'
spotcon      0.9      'free'          0         1       U  # Should be between 0 and 1
spotrad      0.2      'free'          0         1       U  # Should be between 0 and 1
spotlat      0        'free'          -90       90      U  # Should be between -90 and 90
spotlon      0        'free'          -90       90      U  # Should be between -90 and 90 to be on observer-facing hemisphere
spotcon1     0.95     'free'          0         1       U  # Only use for starry! fleck cannot have different contrasts for each spot
spotrad1     0.1      'free'          0         1       U  # Should be between 0 and 1
spotlat1     10       'free'          -90       90      U  # Should be between -90 and 90
spotlon1     45       'free'          -90       90      U  # Should be between -90 and 90 to be on observer-facing hemisphere
--------------------
** Systematic variables **
Polynomial model variables (c0--c9 for 0th--3rd order polynomials in time); Fitting at least c0 is very strongly recommended!
Exponential ramp model variables (r0--r1 for one exponential ramp, r2--r3 for a second exponential ramp)
GP model parameters (A, m for the first kernel; A1, m1 for the second kernel; etc.) in log scale
Step-function model variables (step# and steptime# for step-function model #; e.g. step0 and steptime0)
Drift model variables (xpos, ypos, xwidth, ywidth)
--------------------
c0          1.015           'free'          1           0.05        N
c1          0               'free'          0           0.01        N
A           -15             'free'          -25         0           U
m           -4              'free'          -7          0           U
-----------
** White noise **
Use scatter_mult to fit a multiplier to the expected noise level from Stage 3 (recommended)
Use scatter_ppm to fit the noise level in ppm
-----------
scatter_mult 1.1            'free'         0.8          10           U

I wonder if you also need to specify time_offset_pl1?

I gave that a try, but it looks like the same error:
(eureka) dlong@hikaru obs3 % python run_eureka_t1be_3.py
WARNING: There are 8 metadata save files in the folder: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/
Using the metadata file inside: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7/
and will consider aperture ranges listed there. If this metadata file is not a part
of the run you intended, please provide a more precise folder for the metadata file.
Finished loading parameters from /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7//S4_t1be_3_ap3_bg7_LCData.h5
Starting Stage 5: Light Curve Fitting
Input directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7/
Output directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage5/S5_2024-09-26_t1be_3_run1/ap3_bg7/
Copying S5 control file
Copying S5 parameter control file
Starting Channel 0 of 47
=========================
Starting dynesty fit.
Traceback (most recent call last):
File "/Users/dlong/DataAnalysis/JWST/eureka/t1be/obs3/run_eureka_t1be_3.py", line 31, in
meta = s5.fitlc(eventlabel, ecf_path=ecf_path)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 473, in fitlc
meta, params = fit_channel(meta, time_temp, flux, channel,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 964, in fit_channel
lc_model.fit(model, meta, log, fitter='dynesty')
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/lightcurve.py", line 181, in fit
fit_model = self.fitter_func(self, model, meta, log, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/fitters.py", line 856, in dynestyfitter
start_lnprob = lnprob(freepars, lc, model, prior1, prior2, priortype,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 206, in lnprob
ln_like_val = ln_like(theta, lc, model, freenames)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 85, in ln_like
model_lc = model.eval()
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/Model.py", line 370, in eval
flux = component.eval(channel=channel, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/AstroModel.py", line 400, in eval
starFlux = self.transit_model.eval(channel=chan,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/BatmanModels.py", line 139, in eval
m_transit = self.transit_model(pl_params, time,
File "/Users/dlong/miniconda3/envs/eureka/lib/python3.10/site-packages/batman/transitmodel.py", line 138, in init
self.ds = _rsky._rsky(self.t_supersample, params.t0, params.per, params.a, params.incpi/180., params.ecc, params.wpi/180., self.transittype, self.nthreads)
TypeError: must be real number, not NoneType
(eureka) dlong@hikaru obs3 %
Stage 5 Fit Parameters Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-5-fit-parameters
Name       Value         Free?          PriorPar1    PriorPar2    PriorType
"Free?" can be free, fixed, white_free, white_fixed, shared, or independent
PriorType can be U (Uniform), LU (Log Uniform), or N (Normal).
If U/LU, PriorPar1 and PriorPar2 represent upper and lower limits of the parameter/log(the parameter).
If N, PriorPar1 is the mean and PriorPar2 is the standard deviation of a Gaussian prior.
 See docs for details!
For fleck: can only have one spot contrast (spotcon), spotnpts is the number of temporal points to evaluate at (~200-500)
For starry: assign one spotcon per spot (spotcon, spotcon1, etc), spotnpts is the ydeg of the star map (~30)
------------------
spotstari    90       'fixed'
spotrot      100      'fixed'
spotnpts     300      'independent'
spotcon      0.9      'free'          0         1       U  # Should be between 0 and 1
spotrad      0.2      'free'          0         1       U  # Should be between 0 and 1
spotlat      0        'free'          -90       90      U  # Should be between -90 and 90
spotlon      0        'free'          -90       90      U  # Should be between -90 and 90 to be on observer-facing hemisphere
spotcon1     0.95     'free'          0         1       U  # Only use for starry! fleck cannot have different contrasts for each spot
spotrad1     0.1      'free'          0         1       U  # Should be between 0 and 1
spotlat1     10       'free'          -90       90      U  # Should be between -90 and 90
spotlon1     45       'free'          -90       90      U  # Should be between -90 and 90 to be on observer-facing hemisphere
--------------------
** Systematic variables **
Polynomial model variables (c0--c9 for 0th--3rd order polynomials in time); Fitting at least c0 is very strongly recommended!
Exponential ramp model variables (r0--r1 for one exponential ramp, r2--r3 for a second exponential ramp)
GP model parameters (A, m for the first kernel; A1, m1 for the second kernel; etc.) in log scale
Step-function model variables (step# and steptime# for step-function model #; e.g. step0 and steptime0)
Drift model variables (xpos, ypos, xwidth, ywidth)
--------------------
c0          1.015           'free'          1           0.05        N
c1          0               'free'          0           0.01        N
A           -15             'free'          -25         0           U
m           -4              'free'          -7          0           U
-----------
** White noise **
Use scatter_mult to fit a multiplier to the expected noise level from Stage 3 (recommended)
Use scatter_ppm to fit the noise level in ppm
-----------
scatter_mult 1.1            'free'         0.8          10           U

Hmm, strange - it's possible that the ars and rprs aliases for a and rp (respectively) are broken either in general or for multi-planet fits. I'm quite sure that time_offset_pl1 won't ever be used and that the time_offset parameter doesn't change between planets. While I debug this (likely tomorrow given that my work day is just ending), I recommend you try replacing all instances of ars with a and rprs with rp and see if that resolves the issue. If it does, then I'll know which exact part of the code to patch tomorrow morning

Unfortunately, same error:
(eureka) dlong@hikaru obs3 % python run_eureka_t1be_3.py
WARNING: There are 8 metadata save files in the folder: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/
Using the metadata file inside: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7/
and will consider aperture ranges listed there. If this metadata file is not a part
of the run you intended, please provide a more precise folder for the metadata file.
Finished loading parameters from /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7//S4_t1be_3_ap3_bg7_LCData.h5
Starting Stage 5: Light Curve Fitting
Input directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-26_t1be_3_run1/ap3_bg7/
Output directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage5/S5_2024-09-26_t1be_3_run2/ap3_bg7/
Copying S5 control file
Copying S5 parameter control file
Starting Channel 0 of 47
=========================
Starting dynesty fit.
Traceback (most recent call last):
File "/Users/dlong/DataAnalysis/JWST/eureka/t1be/obs3/run_eureka_t1be_3.py", line 31, in
meta = s5.fitlc(eventlabel, ecf_path=ecf_path)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 473, in fitlc
meta, params = fit_channel(meta, time_temp, flux, channel,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 964, in fit_channel
lc_model.fit(model, meta, log, fitter='dynesty')
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/lightcurve.py", line 181, in fit
fit_model = self.fitter_func(self, model, meta, log, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/fitters.py", line 856, in dynestyfitter
start_lnprob = lnprob(freepars, lc, model, prior1, prior2, priortype,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 206, in lnprob
ln_like_val = ln_like(theta, lc, model, freenames)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 85, in ln_like
model_lc = model.eval()
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/Model.py", line 370, in eval
flux = component.eval(channel=channel, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/AstroModel.py", line 400, in eval
starFlux = self.transit_model.eval(channel=chan,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/BatmanModels.py", line 139, in eval
m_transit = self.transit_model(pl_params, time,
File "/Users/dlong/miniconda3/envs/eureka/lib/python3.10/site-packages/batman/transitmodel.py", line 138, in init
self.ds = _rsky._rsky(self.t_supersample, params.t0, params.per, params.a, params.incpi/180., params.ecc, params.wpi/180., self.transittype, self.nthreads)
TypeError: must be real number, not NoneType
(eureka) dlong@hikaru obs3 %
Stage 5 Fit Parameters Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-5-fit-parameters
Name       Value         Free?          PriorPar1    PriorPar2    PriorType
"Free?" can be free, fixed, white_free, white_fixed, shared, or independent
PriorType can be U (Uniform), LU (Log Uniform), or N (Normal).
If U/LU, PriorPar1 and PriorPar2 represent upper and lower limits of the parameter/log(the parameter).
If N, PriorPar1 is the mean and PriorPar2 is the standard deviation of a Gaussian prior.
 See docs for details!
For fleck: can only have one spot contrast (spotcon), spotnpts is the number of temporal points to evaluate at (~200-500)
For starry: assign one spotcon per spot (spotcon, spotcon1, etc), spotnpts is the ydeg of the star map (~30)
------------------
spotstari    90       'fixed'
spotrot      100      'fixed'
spotnpts     300      'independent'
spotcon      0.9      'free'          0         1       U  # Should be between 0 and 1
spotrad      0.2      'free'          0         1       U  # Should be between 0 and 1
spotlat      0        'free'          -90       90      U  # Should be between -90 and 90
spotlon      0        'free'          -90       90      U  # Should be between -90 and 90 to be on observer-facing hemisphere
spotcon1     0.95     'free'          0         1       U  # Only use for starry! fleck cannot have different contrasts for each spot
spotrad1     0.1      'free'          0         1       U  # Should be between 0 and 1
spotlat1     10       'free'          -90       90      U  # Should be between -90 and 90
spotlon1     45       'free'          -90       90      U  # Should be between -90 and 90 to be on observer-facing hemisphere
--------------------
** Systematic variables **
Polynomial model variables (c0--c9 for 0th--3rd order polynomials in time); Fitting at least c0 is very strongly recommended!
Exponential ramp model variables (r0--r1 for one exponential ramp, r2--r3 for a second exponential ramp)
GP model parameters (A, m for the first kernel; A1, m1 for the second kernel; etc.) in log scale
Step-function model variables (step# and steptime# for step-function model #; e.g. step0 and steptime0)
Drift model variables (xpos, ypos, xwidth, ywidth)
--------------------
c0          1.015           'free'          1           0.05        N
c1          0               'free'          0           0.01        N
A           -15             'free'          -25         0           U
m           -4              'free'          -7          0           U
-----------
** White noise **
Use scatter_mult to fit a multiplier to the expected noise level from Stage 3 (recommended)
Use scatter_ppm to fit the noise level in ppm
-----------
scatter_mult 1.1            'free'         0.8          10           U

Huh, strange - I just tested the multi-planet fitting code on one of my own datasets and it works just fine. Can you please confirm that all of your plots out of Stage 4 all look reasonable?

They all look good. As another check I just ran stage 4 again, tightening up the wavelength range a little and setting it for a single channel. The resulting light curve is identical to the white LC, as expected. Stage 5 still fails with the same error:
(eureka) dlong@hikaru obs3 % python run_eureka_t1be_3.py
WARNING: There are 14 metadata save files in the folder: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/
Using the metadata file inside: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-27_t1be_3_run3/ap3_bg7/
and will consider aperture ranges listed there. If this metadata file is not a part
of the run you intended, please provide a more precise folder for the metadata file.
Finished loading parameters from /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-27_t1be_3_run3/ap3_bg7//S4_t1be_3_ap3_bg7_LCData.h5
Starting Stage 5: Light Curve Fitting
Input directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage4/S4_2024-09-27_t1be_3_run3/ap3_bg7/
Output directory: /Users/dlong/DataANalysis/JWST/eureka/t1be/obs3/Stage5/S5_2024-09-27_t1be_3_run5/ap3_bg7/
Copying S5 control file
Copying S5 parameter control file
Starting Channel 0 of 1
=========================
Starting dynesty fit.
Traceback (most recent call last):
File "/Users/dlong/DataAnalysis/JWST/eureka/t1be/obs3/run_eureka_t1be_3.py", line 31, in
meta = s5.fitlc(eventlabel, ecf_path=ecf_path)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 473, in fitlc
meta, params = fit_channel(meta, time_temp, flux, channel,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/s5_fit.py", line 964, in fit_channel
lc_model.fit(model, meta, log, fitter='dynesty')
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/lightcurve.py", line 181, in fit
fit_model = self.fitter_func(self, model, meta, log, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/fitters.py", line 856, in dynestyfitter
start_lnprob = lnprob(freepars, lc, model, prior1, prior2, priortype,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 206, in lnprob
ln_like_val = ln_like(theta, lc, model, freenames)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/likelihood.py", line 85, in ln_like
model_lc = model.eval()
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/Model.py", line 370, in eval
flux = component.eval(channel=channel, **kwargs)
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/AstroModel.py", line 400, in eval
starFlux = self.transit_model.eval(channel=chan,
File "/Users/dlong/Eureka/src/eureka/S5_lightcurve_fitting/models/BatmanModels.py", line 139, in eval
m_transit = self.transit_model(pl_params, time,
File "/Users/dlong/miniconda3/envs/eureka/lib/python3.10/site-packages/batman/transitmodel.py", line 138, in init
self.ds = _rsky._rsky(self.t_supersample, params.t0, params.per, params.a, params.incpi/180., params.ecc, params.wpi/180., self.transittype, self.nthreads)
TypeError: must be real number, not NoneType

And just as a quick check, I changed the fitter from Dynesty to LSQ. Same error.

I think I found it! In my epf file I had 't0_pl11' instead of 't0_pl1'. Every time I looked at it my brain mushed all of those ls and 1s together, apparently.

Hah yeah that'd do it! Glad you were able to sort it out!


Create LICENSE

Create WTFPL License.

Thanks for the contribution. This was added by pull request #1 .


1.0.0 Cannot read property 'y' of undefined

After upgrading. When hovering, and the tooltip would normally not be visible in it's default placement, I get a ton of errors.

The tooltip is also destroyed until a refresh.
Some variables values if it helps:

0.9.8 works, but the tooltip is not visible if it clips out out view (which I suppose is what you tried to fix in 1.0.0)

Fixed in v1.0.2

Sweet, thanks! Works fine now :smile:


Export and import ILA Portable

Is your feature request related to a problem? Please describe.
Would be more accessible for users, especially with the synthesis engine.
Describe the solution you'd like
Export an archive-able file (either human readable or machine readable) for ILA models.
Describe alternatives you've considered
JSON input parser and the output engine has pretty great supports.
Additional context
N/A

Need to be humane readable.

cb8efba00703fddfe91c3ca00b57db62e8302f85


Conditionally exclude the socket layer to port to other network stacks

I am working on porting some wayland based code to a new platform (using webassembly and the browser); the network layer is necessarily different, and there's no way CLOEXEC sockets will be supported. I'd like to re-use wayland-commons however, so I'd love to get this conditional compilation included, especially considering wayland-protocols depends on it.

So, there are a few things to keep in mind here:

First, once #381 is finished (I don't know when), wayland-commons will no longer exist has a part of the wayland-rs set of crates, meaning the current published 0.29 will effectively go unmaintained.
Second, when trying to port Wayland-based code to a new platform, before event thinking about cloexec sockets, there is the question of ancillary data. Will your network stack support the passing of file descriptors onto your sockets, like unix sockets do? This mechanism is quite core to the Wayland protocol.

Makes sense! I'm actually mostly after wayland-protocols more than wayland-common, but there's currently a dependency there.
Yes, the platform will support fd passing.

Coming back to this, porting the whole thing to other network stacks would require either making the whole wayland-rs crates generic on the network stack used, or include support for those network stacks directly in wayland-rs. In both cases this is some pretty serious change to the library, not simply adding a cargo feature.

Yeah. I was hoping to build my own network layer at this point, but re-use the protocol definitions.

Well, on a positive note, #381 will introduce a wayland-backend crate that separates pretty well the protocol definitions from the wire protocol itself, that crate might be a relevant place to merge support for alternative transport layers.

Now that the 0.30 rework is merged, I'm going to close this PR. Please see the current master branch as a starting point if you are still interested in porting the wayland stack to other platforms. :slightly_smiling_face:


stick with openssl 1.0.x

fix #44
disclaimer: more magic!
https requests for tiles are working for us

HTTPS works now on Android - I recommend merging this :-)


Profile - The photo with a gray background does not appear

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Issue found when executing #10235
Action Performed:

Go to URL https://staging.new.expensify.com/
Click on the avatar picker and upload an avatar
Be ready to turn off your device wifi very quickly, click on the avatar picker and upload an avatar, but IMMEDIATELY turn off your device WiFi before avatar can finish uploading.

Expected Result:
The avatar should become grayed out if the Internet connection is turned off during loading
Actual Result:
The photo with a gray background does not appear
Workaround:
Unknown
Platform:

Where is this issue occurring?

Web
iOS
Android
Desktop App
Mobile Web

Version Number: v1.1.94-4
Reproducible in staging?: Y
Reproducible in production?: N (New feature)
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
https://user-images.githubusercontent.com/43996225/187543212-8144334b-cf84-490b-b04a-71b803408cf0.mp4
https://user-images.githubusercontent.com/43996225/187543221-c7785b5c-013e-419b-b1c0-b045b1ae39f3.mp4
Expensify/Expensify Issue URL:
Issue reported by: Applause internal team
Slack conversation:
View all open jobs on GitHub

Taking a look since this was related to my PR

So I think in the OP they're not actually turning off the device wifi fast enough, you need to do it within a half second otherwise the upload will complete:
https://user-images.githubusercontent.com/31285285/187614879-90351ac6-3564-4c0e-8fa2-c702340d8067.mov
Closing this out


The second word was not actually being nulled. It just happens that the value of the word "poet" is equal to 0x539, which is the flag for the monero currency.
Should be fixed now, the flag on the second word is put back after parsing the seed.


Implement ExtEvent Loop

+1
Waiting for: client(SSH2 kraken) <-> websocket server(gos websocket, ratchet, react) <-> ssh connection.
Can not broadcast on('data'), have to wait for command to complete.


Add XML Storage

the tricky thing with XML storages is that there needs to be some kind of mapping or xsd,
because xml structures are (always) different.
anyone an idea?

implemented using the option for custom storages
thats the only thing that can be done with such flexible storages


[WIP] Acquire presence in sessions

This is currently a big WIP for scraping data from presence in sessions for the current mandate. Not sure how to best structure so that it better fits the purpose of this package and because of that I'd love some feedback about it before I go on a tangent on a refactoring session 😄

Notebook with reasoning behind the script: https://gist.github.com/fgrehm/46df3655c803a214f1ce56d9895bbceb
Initial analysis: https://gist.github.com/fgrehm/d134366a8d135e365cce94554c25b879
Datasets:

2016-12-21-deputies.xz.zip
2016-12-21-presences.xz.zip
2016-12-21-sessions.xz.zip

Related to https://github.com/datasciencebr/serenata-de-amor/issues/137 as well

@Irio @cuducos what do you guys think about having those datasets uploaded to S3 while we discuss how to refactor this hacky script into something that can be reused? This has a lot more data about presence of deputies in Brasilia than the speeches dataset 💭

if the script works, I vote for let's merge it and for let's upload the datasets to S3. Can you check why GitHub says there's conflict? It's just one file, I don't get it…
And, finally: wanna pair to refactor it?

if the script works, I vote for let's merge it and for let's upload the datasets to S3

Up to you guys 😄

Can you check why GitHub says there's conflict? It's just one file, I don't get it…

I don't see any conflict on my end 😕

And, finally: wanna pair to refactor it?

Sure, how about a screenhero session next week? Lets chat over telegram to schedule a good time for both of us

if the script works

I haven't run it in isolation but it is pretty much a copy & paste from the notebook, I can give it a try next week to double check before we pair on it

@Irio: can you upload the datasets to S3 and drop a line so I know I can merge?
@fgrehm: don't worry about conflicts then, I'll figure it out. Drop a line on Telegram and we pair next week then! Many thanks ; )

Just uploaded them to Amazon.
data/2016-12-21-sessions.xz
data/2016-12-21-deputies.xz
data/2016-12-21-presences.xz

Hi Folks
I recently found out about the serenata project, well done project, congratulation to all members ;)
I've been working with the same dataset by myself in the last couple days, my goal was to put everything behind REST API ( biography, speeches, voting, presences on the house, presences on commissions )
It is still work in progress but usable at some extent, following few examples:
Get biography by id
https://brasil-como-vota-api.herokuapp.com/deputados?id=eq.66c341969f0ef1a00ffd66c4b1b3f325d5ca94fb
Get projection of fields from presence in the house
https://brasil-como-vota-api.herokuapp.com/presenca_plenario?select=deputado->>nome,frequencia,data,sessao,legislatura&limit=10
Find voting data filtering by name
https://brasil-como-vota-api.herokuapp.com/votos?deputado->>nome=like.POPÓ&select=deputado->>nome,sessao,proposicao:proposicao->>codigo,voto,data
The dataset is exposed by postgrest.

@thiagofelix that's nice! mind sharing your thoughts on how we can use it as a separate issue for discussion? this is a closed PR and not many people will find it 😄 your API should make scraping a lot easier and more reliable (I've seen some of those endpoints return a 50x quite frequently on business days)
Also, is the code for scraping that data up somewhere?

@fgrehm Thanks for the reply. I am finishing few bits before publish the source code. But should be on my GitHub by the end of week.
I will open another issue once I get the API and source code ready to use so we can talk more =)


Investigate Wrong Head Ratio metric

Describe the bug
With v0.35.0, missed attestations is not a hot issue any more, however we need to figure out why our validator vote for some wrong heads:

Expected behavior
No wrong head

Closing issues for specific incidents on old versions of Lodestar, please re-open if it happens with latest stable


Add Metalsmith plugin to the code page

Hello! I made a Metalsmith plugin for generating a JSON feed from collections, thought it might be a useful addition to your list here.

Thanks!


add internal console logging

Add a menu item to open up an internal console log.

@aleffert please review

:+1: Factor filtering as discussed


Render problem when  there is a line below the table

Hi,@vsch.
I found the text：
|a|b|c|
|---|---|---|
|1|2|3|
---

It can't be rendered because of the diving line
Could you solve this problem？

@Hxuhao233, this happens because table processing occurs right before inline processing and only on paragraph blocks.
The --- causes the previous lines to be interpreted as a level 2 heading and  === would cause it to be a level 1 heading. These are the only character sequences that would cause the table not to be recognized since these are the only post fix markers for preceding lines.
The old implementation of tables in the flexmark-ext-gfm-tables module parses the lines during  block recognition and will parse the above markdown as you expect. However, it has more limitations in table parsing and will only parse tables with GFM limitations.
The only solution for the flexmark-ext-tables module is to include a blank line between the table and the following text if it can be interpreted as a level 1 or level 2 heading.

Sorry,@vsch, I was so late to reply.
So,i should include a blank line to render it correctly.
I get it.Thank you!


Javascript error after app closed

Your description here

Saw an newer version was available and installed it yesterday on my laptop.
I left the app open and only closed it this morning.
Did not take note of how long after closing the app, but since then getting the above Javascript error a few times.
Expected outcome
What should happen?
No error message
Actual outcome
What happened?
Versions
App: 10.0.12
Platform: Win 19
Chrome 64.0.3282.140 (Official Build) (64-bit)
Steps to reproduce

Installed
App Left open & laptop hibernate
Close app
Dont know if these steps are worth anything?

Hi,
this report is for ARC desktop app and should be reported here: https://github.com/advanced-rest-client/arc-electron/
However, it is already fixed. I have released an update a minute ago. See https://github.com/advanced-rest-client/arc-electron/issues/50


Fix CI publish staging workflow

The CI pipeline works for the default workflow. But it fails for the publish_staging workflow when executing the dApp end-to-end tests. The only obvious difference for the tests between both pipelines is that one runs with an e2e mode build of the dApp, while the other uses the staging mode. That might have an impact. But a first analysis reveals an issue that pops up way earlier in the test flow. In fact Cypress fails to load the dApp at all. It can't even access the very landing page.
This is especially awkward for two reasons. First does Cypress initially check if there is a server running under the configured URL. If there is no process listening, Cypress will fail even earlier and reports this specific error. Second, I re-run the job in CI with an SSH connection to the machine. I turned on the logging (set -x) for the run-cypress.sh script. Everything looks fine. And if I curl the dApp while the Cypress tests are running, I successfully get the index.html file. So after all the web-server is working and it serves the build staging version of the dApp. Though something in the setup phase of the VueJS application loaded in the index.html does not seem to work. Unfortunately we are still unable to see the console of the dApp within the videos of the tests. Therefore we need to check if we can reproduce the issue locally with a staging build and then debug it. We might need to add some missing environment variables to the mode.

I filled the details into the issue body.

Okay, the issue is simple: the staging build gets build to working under the sub-path /staging/.... The end-to-end test server serves everything under root /. This means the production pipeline should not be affected.
The question is a little tricky. We somehow need to know for which mode the dApp was build then source their build environment variables to then adopt the server asset path.


Error: PhantomJS timed out, possibly due to a missing QUnit start() call

I'm getting an issue when setting up a simple qunit test to be run through grunt with phantom:
PhantomJS timed out, possibly due to a missing QUnit start() call
I'm kind of stumped, I would love to know what I'm missing!
My grunt file looks like this:
    qunit: {
      all: ['tests/*.html']
    },
    connect: {
        server: {
            options: {
                port: 3000,
                base: '.'
            }
        }
    },

I'm using these versions:
<EMAIL_ADDRESS><EMAIL_ADDRESS>And where is what my sample looks like:

  <meta charset="utf-8">
  <title>Tests</title>
  <link rel="stylesheet" href="/tests/resources/qunit-1.10.0.css">

I know it's running because I get the console log output. Any help would be much appreciated!

@jansepar this fixed the problem for me too! Thx for posting!

I am still seeing same error.  This is my grunt file
 qunit: {             all: {                 options: {                     urls: [                         'http://localhost:8000/test/unit/qUnit.html',                     ]                 }             }         },         connect: {             server: {                 options: {                     port: 8000,                     base: '.'                 }             }         },

var triggerSpy = sinon.spy(mediator, 'trigger');
            contentView.uploadModal.$el.on('hidden', function() {
                setTimeout(function() {
                    QUnit.start();
                    ok(triggerSpy.calledWith('alert:success', 'test success'),
                        'Success message should get displayed');
                    ok(triggerSpy.calledWith('alert:warning', 'Some error message'),
                        'Warning message within the pre tags should get displayed');

                    triggerSpy.restore();
                    contentView.close();
                    contentView = null;
                }, 1000);
            });
            contentView.uploadModal.trigger('fileUpload:success', {
                response: '
 Some error message
'
            });

QUnit.start(); not working
Giving following error- PhantomJS timed out, possibly due to a missing QUnit start() call


Add react-native-community eslint

See https://github.com/facebook/react-native/tree/master/packages/eslint-config-react-native-community.

Hi. Can I have a go at it?

Added functionality in pull request #56.


[zh]Sync content/zh/examples/controllers/fluentd-daemonset.yaml

One of files in this PR is conflicted with #33681.

#33681

Please rebase so that we can review and merge your change to another file.


[WIP] Add support for mapping local variable names

Add support to the mapper to transfer local variables between versions
Depends on #1907
Mapping from 168->167->168 gave one local variable collision. I'm not sure why.
168 -> 169 was the same. I will do more testing on this later and fix this bug.

Closing because this is no longer relevant.


feat!: rename function.function in function.archive

Part of FL-1160
Description

How to test

Screenshots

Notes for developers and reviewers:

Think to update CHANGELOG.md before merge if needed !

/e2e --tests=frontend

End to end tests: :hourglass_flowing_sand: STARTING


Update version to 0.6.0

Creating PR to bump the version number as per the releasing guide
What's Changed

Update check-update to pull from metadata.json by @misscoded in https://github.com/slackapi/deno-slack-hooks/pull/41
[HERMES-4040] Support runtime 0.4.0 by @mcodik in https://github.com/slackapi/deno-slack-hooks/pull/40

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Does not support react native

Can be reproduced with create-react-app, simply having
import {Base64} from "js-base64"

will give
Failed building JavaScript bundle

Think they are the problem:
https://github.com/dankogai/js-base64/blob/master/base64.js#L201-L205

I had a similar problem, but to me the offending lines where these:
https://github.com/dankogai/js-base64/blob/master/base64.js#L18-L22

+1
Problem persists on version 2.4.3

It used to work on version 2.3.2

Try adding:
 "babel-preset-es2015": "^6.24.1",
 "babel-register": "^6.26.0"

To the dependency of your project.  FYI That's on devDependencies of package.json.

Just tried ^
Still gets Failed building JavaScript bundle using create-react-native-app
My package.json

{
  "name": "ioi",
  "version": "0.1.0",
  "private": true,
  "devDependencies": {
    "jest-expo": "25.0.0",
    "react-native-scripts": "1.11.1",
    "react-test-renderer": "16.2.0"
  },
  "main": "./node_modules/react-native-scripts/build/bin/crna-entry.js",
  "scripts": {
    "start": "react-native-scripts start",
    "eject": "react-native-scripts eject",
    "android": "react-native-scripts android",
    "ios": "react-native-scripts ios",
    "test": "node node_modules/jest/bin/jest.js"
  },
  "jest": {
    "preset": "jest-expo"
  },
  "dependencies": {
    "babel-preset-es2015": "^6.24.1",
    "babel-register": "^6.26.0",
    "expo": "^25.0.0",
    "js-base64": "^2.4.3",
    "react": "16.2.0",
    "react-native": "0.52.0"
  }
}

Error:

Then just add buffer to your package.json.
ex. https://github.com/facebook/react-native/issues/14796

Thx. Adding buffer solved my problem. Works with or without babel-preset-es2015 and babel-register


RxFileMonitor (Mac)

https://github.com/CleanCocoa/RxFileMonitor
Put this up to get reactive file system events when monitoring a folder (or file). Pretty simple thing, really.

Very cool, what a neat idea for a framework! Here's an issue where we've moved a project over to the org, if that's something you're interested in. I've invited you to the org, let me know if I can be of further assistance :bow:

Will look into the checklist. I've got a small favor to ask, though: can you point me to naming conventions in the land of reactive/declarative programming? I was told FileEvents.folder(...) would be a better fit; I now see that Monitoring.folderMonitor  carries my imperative training with it :) My next suggestion would be Monitor.folderMonitor(url:).

Why even have the class method? Could you not just init the FileMonitor instance?

Yeah, either makes sense to me 👍 I prefer the instance like Tony suggests.

By making the monitor type a subclass of Observable, that is? I cannot imagine the consequences, though. Could work, if you say that's idiomatic :)

Here we go:

[ ] replace copyright with "Copyright (c) RxSwiftCommunity" in code
[ ] replace copyright in README
[ ] replace copyright in LICENSE
[ ] update the remote URL of the Podspec (if any)
[ ] update CI badge
[ ] update CI settings
[ ] transfer repository ownership (once you're a contributor)
[ ] fork the repo back to your personal account
[ ] add project to https://github.com/RxSwiftCommunity/rxswiftcommunity.github.io

@ashfurrow I thought I'd have write access after transferring ownership, but no :) Can you merge the PR or grant me write access to https://github.com/RxSwiftCommunity/RxFileMonitor to continue?

Should be good now, sorry about that!

Done with the migration, now comes integrating your feedback 👍


Syntax highlighting of shaders

While working on large shaders as in #628, it would help to have WGSL syntax highlighting. Right now the shaders are strings, which makes this hard.

Fix via an IDE: there is a VSCode extension to highlight WGSL in strings in JS, but not for Python.
Fix by moving (most) shaders into their own files.

Related https://github.com/pygfx/shadertoy/issues/5

We now have our shaders as .wgsl files, and we have https://github.com/pygfx/python-string-shaders (a VSCode extension that is also aware of our templating syntax).


ComponentStatusRule conflicts

We have our own ComponentMetadataRule that sets the status to "milestone" when matching particular patterns, such as "RC".
This is conflict with the new ComponentStatusRule that was added to this project that sets the status to "integration".
While I appreciate the recognition that latest.release is largely unusable due to artifacts published without this information, the global application of ComponentStatusRule seems problematic since it is encroaching into the territory of opinion/preference. With capabilities, this is more clear cut and less problematic, so the inability to disable those rules seems reasonable.
However, since this is being applied to any and all artifacts, I would ask for an option to disable this rule.

Thank you for this feedback @ko-at-work. I was mostly thinking about components not being release without thinking too much about other statuses they may get. From the perspective you shared, I agree that it is likely too opinionated to be applied everywhere. I might remove the automatic applying altogether but keep the rule implementation itself so users can use it as a convenience if they want.
One thing I wonder though: Can't you add your rule "on top" of the plugin? Then your rule should override what this rule is doing.

Thank you for elaboration @ko-at-work. Much appreciated. In hindsight, I should have not added an all rule, no matter what it is about. I turned the rule into a custom rule  and made the "markers" configurable. That's something I would like to use in some projects, but this way it is completely optional.


Create a rubygem

Would it be possible to add a .gemspec and release this on rubygems.org? The ability to gem install this would be very useful!

@jsok - I see you created a gem on Rubygems - but it doesn't currently appear to be in sync with the module on the Forge? Is there any way we can make sure that changes here are inline with those in rubygems?


Segmentations converted from nii to dcm not rendered in OHIF Viewer

Hello,
I am having issues rendering dcm segmentations in OHIF Viewer(version number : 4.12.30).
The segmentations come from ProstateX IDC collection, converted to nii using dcmqi, then converted back to dcm using dcmqi(v 1.2.5) again.
The dcm file originating from IDC buckets is rendered properly into OHIF, however the dcm file from dcmqi conversion from nii format is not rendered properly.
Here is the error reported by the OHIF Viewer :

To reproduce this issue, please see this ipynb.
Additional info :

IDC bucket dcm

metadata
dcm file

DCMQI converted dcm(nii -> dcm)

metadata
dcm file

Diff between two dcm metadata files

Thank you!
@fedorov

Thanks for the report.  I think it's related this code: https://github.com/dcmjs-org/dcmjs/pull/220 but with your new data the error is larger than the (somewhat arbitrary) tolerance doesn't account for the differences due to numerical issues in your round trip.
You can see from the diff you provided that the orientation numbers are similar but not close enough to be considered matching by OHIF.
-       (0020, 0037) Image Orientation (Patient)         DS: [1.000000e+000, -1.990110e-010, 4.961899e-011, 2.051034e-010, 9.702957e-001, -2.419219e-001]

+       (0020, 0037) Image Orientation (Patient)         DS: [1, 3.04622104e-12, 2.48094791e-11, 3.04622104e-12, 0.970295727, -0.241921902]

This is probably because nifti format uses 32-bit floats for orientation while DICOM uses a decimal string which is closer to a double (64 bit float), like the Number type in JavaScript.
Since nifti is used a lot I guess we'll end up needing to "dumb down" ohif to consider anything that's within 32-bit float rounding error to be close enough.
Heres' some more background on issues related to DS in dicom.
https://github.com/dcmjs-org/dcmjs/issues/175
https://github.com/pydicom/pydicom/issues/1632
https://stackoverflow.com/questions/32631178/writing-ieee-754-1985-double-as-ascii-on-a-limited-16-bytes-string

After a bit more research @ccosmin97 and I determined that there is a nifti2 with a 64 bit float option, but apparently it's not widely used.  We confirmed dcmqi is writing out a nift1 32-bit header and it's not clear that ITK has an option to write a nifti2 header or that the nnunet code would be able to read a nifti2 header.  He will confirm if this really is a 32-bit vs 64-bit truncation issue or whether it is an issue with trying to overlay the SEG on a series that really does have different orientation.  If it's a truncation issue we can look into making OHIF/dcmjs less strict about matching orientations.

The SEG and the original MRI data do have the same orientation(the SEG references to the T2 modality) :

The SEG has green value for foreground and yellow for background, it overlaps with the T2 data.
Besides, there seems to be no numerical rounding issues of the Image Orientation Patient for the .nrrd file conversion from dcm SEG using dmcqi v1.2.5 compared to the T2 data(DS: [1.000000e+000, -1.990110e-010, 4.961899e-011, 2.051034e-010, 9.702957e-001, -2.419219e-001]):

Finally, when converting back the nrrd segment files back to dcm using dmcqi v1.2.5, and then uploading to OHIF, it works fine, no errors are reported, as opposed to doing the same process with nifti segment files.
Since the converted nifti files are using a nifti1 header, it might explain the rounding issues that are not accepted by OHIF.
The same behavior regarding nifti files issue with OHIF upload can be observed with different ProstateX PatientIDs.
Please find attached this folder containing log files, PatientID == 0004 dcm data(T2,ADC,SEG_T2) and dcm metadata comparison.
This ipynb can be used to reproduce this issue.

We hit that rounding issue so many times now, and from what I recall last time, the tolerance was lowered to 0.01. When I looked at the data with Cosmin, I did not see the differences of significance.
See here: https://github.com/OHIF/Viewers/issues/2656#issuecomment-1014682438.
I can't believe we are exceeding 0.01 tolerance while writing those segmentations. Are we? I don't think I see that in the example above.
Fortunately, @Punzo is back!

Hi, the tolerance as described in https://github.com/OHIF/Viewers/issues/2656#issuecomment-1014682438 is actually 0.001. I will double check asap if it's indeed a tolerance issue.

@ccosmin97 can you kindly provide also the MRI, please? the studyID will be enough, thanks!

Ah just seen that you already provided everything here https://www.dropbox.com/sh/vq3oe7f2fwl5p4r/AACNbGeKfkOQSEpzEXC3OlUua?dl=0

found the issue here: https://github.com/dcmjs-org/dcmjs/blob/master/src/utilities/orientation/nearlyEqual.js#L24 this condition was too strict (Number.EPSILON is around 10^-16). This means in the case of comparing two numbers close to zero (e.g., around 10^-[11:13]), these were not considered close enough to zero. Therefore we were using relative error comparison which does not work when both numbers are close to zero.
I fixed the issue changing the cost value Number.EPSILON (10^-16) to epsilon^2 (input parameter, default is epsilon = 0.001, for OHIF-IDC is 0.01). See https://github.com/dcmjs-org/dcmjs/pull/304

@pieper we discussed this further, and @Punzo and I agreed that in order to address immediate needs in this issue, it makes sense to merge https://github.com/dcmjs-org/dcmjs/pull/304 first. Effectively, that PR makes the two branches of the condition statement consistent in using the same tolerance, which one could think of as a bugfix.
Going forward, we absolutely have to rethink how those tolerances should be defined in a more meaningful way. What I suggested after you left the meeting Steve, perhaps we could do the following:

in the user interface indicate what is the currently used tolerance and (more importantly) what are the implications of using this tolerance for the specific image (e.g., given the extent of the image, calculate the maximum shift in the location of the annotation that is affected by the tolerance);

allow user to adjust the tolerance to have more or less strict checks, perhaps accompanied by a warning message on the screen when the tolerance is increased beyond the default value.

I am all for a more meaningful handling of these scenarios, but I think we will need time and iterations to arrive at that ideal solution. In the meantime, based on my understanding and discussions with Davide, I believe the PR mentioned above is a valid bugfix and will allow addressing immediate need for IDC use case development.

I just wanted to confirm that the OHIF update with the condition relaxation fixed this issue with the same data mentioned above(idc dcm -> nifti -> dcm) for PatID004, ProstateX collection:

Thank you @Punzo @fedorov @pieper !!

@ccosmin97 please note tahht if you are in OHIF version  4.12.41 (however this version did not hit yet the IDC production portal, you can check the version iin thhe Debug Info window), you can use the run time tol value for segs. Please have a look at video: https://github.com/OHIF/Viewers/pull/2924#issue-1369720769

@ccosmin97 please note that if you are in OHIF version 4.12.41 (however this version did not hit yet the IDC production portal, you can check the version in the Debug Info window), you can use the run time tol value for segs. Please have a look at video: #2924 (comment)

On this note, after changing the runtime tolerance to 1e-1(instead of default 1e-2), the segmentations objects were correctly loaded, reverting back to 1e-2 or lower fails to load the segs on the right hand panel :
https://user-images.githubusercontent.com/72577931/193093510-ec8d86bb-7598-4dd6-b246-632cc9041ae3.mp4

@ccosmin97 for the sake of completeness, can you please share the sample that allows reproducing this issue? Can you confirm this is not the issue if you use dcm2niix for conversion of the input image?

@ccosmin97 for the sake of completeness, can you please share the sample that allows reproducing this issue? Can you confirm this is not the issue if you use dcm2niix for conversion of the input image?

I compared using SimpleITK vs dcm2niix for conversion of MR data which is the input for nnunet prediction segmentations.
The problematic cases throwing an error in OHIF are(t = timestamp / 2 studies for each patient/ qin-prostate-repeatability) : PCAMPMRI-0015-T1, PCAMPMRI-0007-T1, PCAMPMRI-0013-T2, PCAMPMRI-0009-T1, PCAMPMRI-00007-T1, PCAMPMRI-00006-T2.
The notebooks are here : sitk.ipynb and dcm2niix.ipynb.
The only difference between these 2 ipynbs is the use of dcm2niix or sitk for dcm data to nii conversion for the input data to the nnunet segmentation framework.
After looking at OHIF, every patIDs(cf problematic cases) nnunet seg objects is correctly rendered except PCAMPMRI-0013-T2 if we use dcm2niix instead of sitk.
(left/right : dcm2niix conversion for patID15/sitk conversion for patID15)

dcm2niixversion used : Chris Rorden's dcm2niiX version v1.0.20220505  GCC7.5.0 x86-64 (64-bit Linux)
v1.0.20220505
SimpleITK version used : 2.2.0
OHIF version : 4.12.41, with default thresholding value

After looking at OHIF, every patIDs(cf problematic cases) nnunet seg objects is correctly rendered except PCAMPMRI-0013-T2 if we use dcm2niix instead of sitk.

Can you parse this a bit for me?  It sounds to me like you are saying dcm2niix is correct but sitk is incorrect?  And you are saying that the issue that the PatientID field is incorrect in the sitk cases?  Can you say exactly what is incorrect?

Cosmin, can you attach both output segmentations, the one from sitk and dcm2niix?

After looking at OHIF, every patIDs(cf problematic cases) nnunet seg objects is correctly rendered except PCAMPMRI-0013-T2 if we use dcm2niix instead of sitk.

Can you parse this a bit for me? It sounds to me like you are saying dcm2niix is correct but sitk is incorrect? And you are saying that the issue that the PatientID field is incorrect in the sitk cases? Can you say exactly what is incorrect?

Sorry @pieper for the confusion, to clarify :
nnunet segmentations obtained from dcm2niix MR data conversion as an input(here T2+ADC dcm->nii) are correctly rendered in OHIF without the 'segmentations orthogonal to the acquisition place of the source data are not supported yet.'
However, converting the same MR data with SITK instead of dcm2niix produces the OHIF loading error mentioned right above.
The patientID field is correct everywhere, sorry again for the confusion, I only mentioned patID for specific cases where using sitk dcm to nii conversion of MR data input shows this OHIF error, which does not happen when using dcm2niix.
This MR data(T2+ADC) is the input of the nnunet segmentation pipeline.
I hope that helps!

Cosmin, can you attach both output segmentations, the one from sitk and dcm2niix?

Here are the output segmentations, in dcm format, for dcm2niix and sitk dcm to nii conversion of MR input.

Thank you @ccosmin97 that clarifies things 👍
Since we have full control over this use case, I would suggest copying the ImageOrientationPatient header values from the inputs to the output dicom files as an option to work around the known limitation of 32 orientation representation in nifti.

I would suggest copying the ImageOrientationPatient header values from the inputs to the output dicom files as an option to work around

I am not sure this is needed, since we already have segmentations that work in OHIF when using dcm2niix.
But what this means it that we/others might get segmentations that will not work with the default and will require adjusting tolerance threshold to (seemingly) unreasonable values.


Support for RSK

Preliminary PR for Rootstock support in SDK.
Added support for Rootstock wallet creation, addresses, and transaction building/signing.

we normally avoid committing changes to the build directory unless it's build time - since it's between tags there's no real need to update it

No need for a createNewWallet since it builds upon an existing wallet and uses separate code paths if the initialised wallet is using the rootstock network.

sorry! https://github.com/blocktrail/blocktrail-sdk-nodejs/blob/2404e55d8b4b1d62ac2ea5c452f47d006f39c6ea/lib/wallet.js#L889 maybe just revert to the array shizzle, seems we keep to that elsewhere


Require doesn't work as expected when using mixed body input

Following this example, the required tag doesn't work as expected.
type UpdateUserInput struct {
	Payload struct {
		Display string `json:"display" in:"required"`
	} `in:"body=json"`
}

If body is not passed in request, no error is thrown (while it should)

Hi @AlessandroSechi, actually httpin won't traverse recursively into deeper levels of the input struct. Which means only the 1st level of the fields tagged with in directives will be executed. This is by design, in order to reduce complexity.
Example:
type FooInput struct {
	Token  string `in:"header=Authorization"` // ok
	Color  string `in:"query=color"`          // ok
	Nested struct {
		Name string `in:"query=name"` // not ok
	}
}

// input
// POST "/foo?name=apple&color=red"
// Header: Authorization: my-secret-here
// output
// &main.FooInput{Token:"my-secret-here", Color:"red", Nested:struct { Name string "in:\"query=name\"" }{Name:""}}

Try it here: https://goplay.tools/snippet/SLZh8JXjMsL
For your snippet above, once httpin has run the body directive on the 1st level field Payload, it's done. The inner fields including Display of Payload won't get touched. This is why the required tag won't work.
IMO, httpin will and should focus on defining the process of data extracting and leave validation related stuff to other Go packages. Here required is really an exception. However, to some extent the required directive is an essential part of httpin as its function is really limited but frequently-used.
For your usecase, I'd like to suggest another package for struct validation: https://github.com/go-ozzo/ozzo-validation.

The directives defined in a nested struct will be executed now, since v0.12.0 (brought in via ggicci/owl@v0.1.17). TODO: update docs.

Awesome! Thanks

@ggicci Thank you for you work.
But both with 0.12.0 and latest 0.14.2 the example in the first message of the thread doesn't work.
{
  "field": "Payload",
  "source": "required",
  "key": "",
  "value": null,
  "error": "resolve field \"Payload (struct { Display string \\\"json:\\\\\\\"display\\\\\\\" in:\\\\\\\"required\\\\\\\"\\\" })\" failed: resolve field \"Payload.Display (string)\" failed: execute directive \"required\" with args [] failed: missing required field"
}

And in source code I don't see any tests of required with json body, only for query and form. Maybe I miss something?

Hi @ashl1, the required directive always works with at least one another directive together, and it checks whether the field is set by a former directive or not, even the final value is a "zero" value. It doesn't run validations like ensuring the value not to be empty or zero. Another limitation is that, the required directive doesn't know if the the field tagged as required (Display) is set/populated by a directive that run by a parent field (Payload). That's why it threw an error.
I admit that the designed behaviour of directive required has brought a lot of confusion. I will reconsider it and make it more clear and easy to use.

Let me explain the design purpose of required directive again here.
The required directive is used to make sure the key is present in the input data. It always works together with other directives, for example:
type Payload struct {
	Name   string `in:"query=name;required"`      // ensure the key "name" is present in querystring
	Age    int    `in:"form=age;required"`        // ensure the key "age" is present in the submitted form
	Secret string `in:"header=x-secret;required"` // ensure the key "x-secret" is present in HTTP header
}

It's not designed to check the final value of the tagged field is empty/zero or not.

A new directive nonzero is introduced since v0.15.0.

A new directive nonzero is introduced since v0.15.0.

Difference between required and nonzero please read https://ggicci.github.io/httpin/directives/required

Difference between required and nonzero please read https://ggicci.github.io/httpin/directives/required


pip-tools crashing while installing

Environment Versions

python:3.9.13-alpine3.15

Steps to replicate

pip install "pip-tools<7.0"
...
...

Expected result
Successful installation
Actual result
Traceback (most recent call last):
40.35   File "/usr/local/bin/pip-sync", line 5, in
40.35     from piptools.scripts.sync import cli
40.35   File "/usr/local/lib/python3.9/site-packages/piptools/scripts/sync.py", line 17, in
40.35     from .. import sync
40.35   File "/usr/local/lib/python3.9/site-packages/piptools/sync.py", line 11, in
40.35     from pip._internal.commands.freeze import DEV_PKGS
40.35 ImportError: cannot import name 'DEV_PKGS' from 'pip._internal.commands.freeze' (/usr/local/lib/python3.9/site-packages/pip/_internal/commands/freeze.py)

This has been fixed in pip-tools v7.0.0.


Deno KV adapter for lucia V3

Feature was requested here but later on closed: https://github.com/lucia-auth/lucia/issues/1177
I created the adapter for a personal project, so might as well share it in case it's helpful for someone else.
First time contributor so sorry if I ended up making more or less changes than needed, I did make sure that the tests were passing.
Let me know if you have any questions.
PS: Thanks for the project!

We're not open to adding additional adapter packages to the repo. I'll likely add a section to share community maintained adapters so if you can self publish it, I can add it here. Thanks!

@adriancova Can you abstract it into a separate package, publish it and reference it here or in new discussion?

Hey @waptik sure, happy to help.
I'll try to have it ready this weekend and will let you know on this thread once it's done


TORQUE-686

The "filter" option was renamed to "selector" to be more consistent and docs updated.
I've already fixed and run the integs tests and everything went ok.

One note - you'll also need to make the same changes on the DSL side as well (torquebox.rb)

Done. Thanks @dobozysaurus!

Merged into 2x-dev - thanks!


Reject null parameter values rather than crash.

What this PR does / why we need it:
Reject in a human-friendly manner as below rather than panic with a stack trace.
$ kubectl kudo install kafka -P a.yaml
Error: could not parse parameters: errors while unmarshaling following keys of the parameter file a.yaml: a has a null value (https://yaml.org/spec/1.2/spec.html#id2803362) which is currently not supported
$ echo $?
255

Fixes #1602

Nice! ship Though we probably need this check on the server-side (IAW) too.

I believe it crashed in the same code. The line numbers and function names differ between the stack traces because the binaries were built at different commits and there was a refactoring in the meantime.


drivers/sensor/stm32_temp: configurable ADC calibration on first read

As some STM32 series (F1, F2, F4, F7, L1) does not support ADC
calibration at all (see drivers/adc/adc_stm32.c file) setting
calibration bit in ADC sequence structure leads to reading failure.
In this particular sensor it means that first temperature readout
after reboot gives always wrong value which is very annoying. To
prevent that additional boolean property has been added to
st,stm32-temp-common.yaml binding which defines if ADC calibration
during first readout should be done or not.

I've removed the calibration in #44052, calibration shouldn't be required for the temp sensor to work. The ADC hardware is already calibrated by the driver on init if I'm not wrong.
Also this driver can gives wrong temp value if something else is samping another channel on the same ADC hardware, which should be fixed by #44052, this is due to the temp sensor driver only adc_channel_setup the channel on init and not right before the read, therefore once the configuration is overwritten by something else, subsequent temp sensor read will get the wrong value.

Not required anymore, closed


cached images have black background

when an image is retrieved from cached the image always has a black background. transparency is lost
first load:

from cache:

Thanks for the report, I saved the image as JPG, and didn't think about the
transparency. I'll make a fix saving bitmaps into PNG and publish a new
version, 1.2 I think
Le 22 avr. 2015 19:36, "Bodo1981"<EMAIL_ADDRESS>a écrit :

when an image is retrieved from cached the image always has a black
background. transparency is lost
first load:
[image: initial_load]
https://cloud.githubusercontent.com/assets/1597580/7280967/cdbe7224-e926-11e4-9017-877ec48a1070.png
from cache:
[image: from_cache]
https://cloud.githubusercontent.com/assets/1597580/7280966/cdbad8d0-e926-11e4-919e-527141570d14.png
—
Reply to this email directly or view it on GitHub
https://github.com/florent37/DaVinci/issues/5.

I fixed it in the 1.0.2, can you try ?

great job. thx


Rotate Supabase Testnet API key and update in github secrets

https://github.com/credbull/credbull-defi/blob/5eaecdf3aa1fe3dc25cc9cb23dbeccca085c4ca0/packages/contracts/resource/testnet.toml#L14

Fixed on dev branch


App Crashes After Building and Installing on a real device

SDK Used: Sdk 46
PLATFORM: Android
After cloning the app using ' npx create-react-native-app --template https://github.com/expo-community/expo-firebase-starter'
I test the app and it works fine on the emulator, but after building the app using EAS, the app seems to crash.
It only shows the loading screen for the app,then it crashes.

Hi @CaptainEboy, does it logs the error on crash?

I ran into similar issues as well when installing it on an Android device
platform: android 12
phone : mi
i attached the error logs below

found the answer
you need to configure the environment variables in expo

its because your local emulator has the environment values but during production, your environment values are not available with expo.