Gen-3 butler tutorial

@DouglasLeeTucker I've started a branch and notebook for us to collaborate on as we learn more about the Gen-3 butler.
This PR is READY to merge

@bechtol here is a new version of the notebook. We should ask @TallJimbo to take a look, and if the syntax is stable enough, I'd suggest merging this into master.

A few comments:

The current definition of repo points at the ci_hsc_gen3 git repo, which is not a Gen3 data repo.  I think things would be much clearer if you just ignored that directory and defined everything relative to the DATA subdirectory within it (which is a Gen3 data repository).
With that in place, I would recommend initializing the Butler instance with just butler = Butler(repo) (assuming repo == os.path.join(os.environ["CI_HSC_GEN3_DIR"], "DATA")).  Passing the full name of the config file works, but is more advanced usage, and since it's usually a required argument I think establishing a convention of passing it positionally is a good idea.
Setting the collections attribute as in cell 19 is not recommended; it skips some checks that happen at Butler construction that would provide better error messages than what you'd otherwise get.  It sounds like I might want to go make that a property to add that checking, but I think it's best to err on the side of caution in general about modifying attributes in Python: if you don't know it's safe, don't do it, because the language has no way of distinguishing for users (even by convention) which attributes are conceptually settable.  In this case (and until that property is operational), I would recommend:

butler = Butler(butler=butler, collections=collections)

(yes, this is the rare case where the first config argument is not necessary, which doesn't help my previous argument about passing it positionally, but I think the case where you don't know what collection you're starting with is unlikely to happen often outside examples, and adding a collections property would take care of that, too).
Other than that last issue, I think the syntax here is all pretty stable.  I would expect the optional arguments of the various query methods to see some minor changes, but probably not the default behavior.

Thanks @TallJimbo I've implemented your suggestions.
Regarding this: "I think the case where you don't know what collection you're starting with is unlikely to happen often outside examples, and adding a collections property would take care of that, too"
Why do you say that it is unlikely to not know what collections you are starting with? It seems like for the immediate future, someone just giving you the path to a repo is fairly common?

Because a Gen2 repo maps more closely to a collection than a Gen3 repo, I think there are going to be very few Gen3 repos in the wild outside of those used in CI or in small-scale development on people's laptops (e.g. perhaps one or two for all of /datasets on NCSA GPFS) .  So when transferring knowledge about processing outputs between people, the repo will generally be implied from context, and the collection will be what they tell you.

I see. So the use case may be more like "I think Keith has been working on something cool, let me see if I can find the collection he was working off of" (the way that I might explore his "work" directory on a shared disk).

@DouglasLeeTucker @bechtol I think this is ready for review.

Yes, good point - that kind of use case is still going to involve butlers without collections.  I'd normally expect that use case to be better supported by command-line tools that do that kind of querying, but I suppose that may not be the case in an environment like the LSP where there's a strong weight towards using notebooks over command-line tools.

I thought there was also no guarantee that the repo will be on the local disk. I guess there will be command line tools for accessing the structure of remote repos?

I guess there will be command line tools for accessing the structure of remote repos?

Yes, exactly.

OK, will look at it soon!  --DLT
--
Douglas L. Tucker
Fermilab                        Tel:    +1-630-840-2267
MS 127                          FAX:    +1-630-840-8274
PO Box 500                      E-mail<EMAIL_ADDRESS>Batavia, IL 60510 USA           http://home.fnal.gov/~dtucker/
From: Alex Drlica-Wagner<EMAIL_ADDRESS>Reply-To: LSSTScienceCollaborations/StackClub<EMAIL_ADDRESS>Date: Friday, April 17, 2020 at 12:58 PM
To: LSSTScienceCollaborations/StackClub<EMAIL_ADDRESS>Cc: Douglas L Tucker<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [LSSTScienceCollaborations/StackClub] Gen-3 butler tutorial (#225)
@DouglasLeeTuckerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_DouglasLeeTucker&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=FNAzyPnzcjNEyqXtzvBQkxXwKOVWLuTvfIKmY1AjRkg&e= @bechtolhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bechtol&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=86Yr4ZFrIfGP6kjvW3rctisuE_0SqaFwLgrX_LEtw4A&e= I think this is ready for review.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LSSTScienceCollaborations_StackClub_pull_225-23issuecomment-2D615384353&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=vpsJ_qd3fT6IcdGi3CIeQHEqAsu-cS1gJcw6Bqsn0po&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AARAHOMUDOVWKMBXXOZFJSLRNCKCNANCNFSM4I5SSKQA&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=kro7HiaDsdS4pc1-BnU9Tg6NjURL9ix0r4C0vmpJUYA&e=.

Dear Alex,
I ran your update of the Gen-3 butler tutorial, and it worked fine!
While running it, though, I found 2 or 3 few very minor typos, all in this one section of text (3 typos if you consider “a butler” instead of “the butler” a typo 😊 , but 2 typos otherwise):
“To create a butler you need to pass it a configuration file and a run name. The run name tells the butler where the place output files. More on Butler configuration  can be found herehttps://pipelines.lsst.io/modules/lsst.daf.butler/configuring.html. By investigating the directory structue, we find that the 'collection' is…”
•
“To create the butler you need to pass it a configuration file and a run name. The run name tells the butler where to place output files. More on Butler configuration  can be found herehttps://pipelines.lsst.io/modules/lsst.daf.butler/configuring.html. By investigating the directory structure, we find that the 'collection' is…”
I am not sure if you fix those typos now.  If not, I can go ahead and OK the pull request.
Thanks!
Best regards,
Douglas
--
Douglas L. Tucker
Fermilab                        Tel:    +1-630-840-2267
MS 127                          FAX:    +1-630-840-8274
PO Box 500                      E-mail<EMAIL_ADDRESS>Batavia, IL 60510 USA           http://home.fnal.gov/~dtucker/
From: Alex Drlica-Wagner<EMAIL_ADDRESS>Reply-To: LSSTScienceCollaborations/StackClub<EMAIL_ADDRESS>Date: Friday, April 17, 2020 at 12:58 PM
To: LSSTScienceCollaborations/StackClub<EMAIL_ADDRESS>Cc: Douglas L Tucker<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [LSSTScienceCollaborations/StackClub] Gen-3 butler tutorial (#225)
@DouglasLeeTuckerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_DouglasLeeTucker&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=FNAzyPnzcjNEyqXtzvBQkxXwKOVWLuTvfIKmY1AjRkg&e= @bechtolhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bechtol&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=86Yr4ZFrIfGP6kjvW3rctisuE_0SqaFwLgrX_LEtw4A&e= I think this is ready for review.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LSSTScienceCollaborations_StackClub_pull_225-23issuecomment-2D615384353&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=vpsJ_qd3fT6IcdGi3CIeQHEqAsu-cS1gJcw6Bqsn0po&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AARAHOMUDOVWKMBXXOZFJSLRNCKCNANCNFSM4I5SSKQA&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=R-57Mm7OJX1IswbY4ezVTQ&m=AJIdw16wQ0BoKPrti_iy0j8P1K9h2YZylYiIxTS3Cr0&s=kro7HiaDsdS4pc1-BnU9Tg6NjURL9ix0r4C0vmpJUYA&e=.

Thanks @DouglasLeeTucker! Caught two of those typos, but I think I'm missing the third. Feel free to change the text directly on github if you like, otherwise I think you can approve the pull request.

Looks good!  Thanks!

For completeness, it might be useful to show how to get the dataIds with the Gen-3 butler.
dataids = [x.dataId for x in registry.queryDatasets('calexp', collections='shared/ci_hsc_output')]
dataId = dataids[0]
calexp = butler.get('calexp', dataId=dataId)

Also:
refs = list(registry.queryDatasets("calexp", collections="shared/ci_hsc_output")
calexp = butler.getDirect(refs[0])

This avoids repeating some lookups that were already done in the original query, and it ensure that the dataset you found is the dataset you load - which might not be the case otherwise if the collections you pass to queryDatasets differ from butler.collections.
Another variant:
dataIds = list(registry.queryDimensions(["exposure", "detector"], datasets=["calexp"], collections="shared/ci_hsc_output"))
calexp = butler.get('calexp', dataId=dataIds[0])

This one does behave like the original in terms of doing a second lookup with possibly-different collections, but the original query is more flexible in that you could provide multiple datasets (requiring an instance of all datasets to be available for that data ID) or ask for different data ID keys than what is used to identify the dataset, which will invoke various built-in relationships.

A few questions:
calexp = butler.get(refs[0])

Also, it would also be nice to show an example of syntax to select a subset of dataIds, e.g., the calexp's for a particular filter

Do I need to use butler.getDirect or is get smart enough to identify that I've given it a dataRef and pass it to getDirect?

get can take a DatasetRef, and it also will check that the one retrieved from the butler's collection is the same one fully identified by the DatasetRef if DatasetRef.id is not None (DatasetRefs can either be in a "resolved" state in which they have an ID that fully identifies them independent of collection, or in a "unresolved" state in which they're no more than a combination of dataset type and data ID).

Tip I just learned is that when processing the ci_hsc_gen3 dataset, the butler is evolving quickly enough that one needs to take some care to align your version of Stack with the version of ci_hsc_gen3. This can be done by checking out the weekly tag of ci_hsc_gen3, e.g.,
git checkout w.2020.16
I suggest that we add this to the description at the top of the notebook, along with the standard line to check what version of the stack you are using, e.g.,
eups list -s | grep lsst_distrib

I guess you just want us to emphasize this again?
The correspondence between the release used to generate the repo and the release used to run the notebook has been discussed extensively (in fact, the repo name explicitly includes the release version). All stack club notebooks should include the release version at the top (as this notebook does), and I think we state clearly in some general instructions that notebooks are not guaranteed to work with other versions.
Gen-3 butler development is moving so rapidly, that there is no way that we are going to keep this notebook at the bleeding edge.
At some point I will get around to merging this PR, but I got stalled because one of the suggestions from Jim did not execute properly (this may again be a version issue).

@bechtol I added an example to query by filter. I am going to merge this PR, but I would not object to someone else continuing to develop this notebook.


fix response array conversion

When deleting media, it always returns 0 event when the file is actually deleted, so I figured out the problem by tracing code and find out array conversion is not happening when retrieving media.

@Mdhesari not sure I get what you mean, there is a check on the next line for determining if the response is an array or not.

@unicodeveloper I think what he is saying, is the API call to destroy doesn't return an array, but rather an ApiResponse object. So the check for is_array will always fail without an explicit cast to an array. thus always returning false even on a sucess.

@brandon14 thanks for clarifying. I understand better now.


Adds support for Select, Scan and Search queries

Changes interval query and response API in order to support the three new queries
Legacy mode for Scan queries is configurable from application.conf
Updates documentation and examples for new queries
Updates unit tests to examine new queries
All Druid Queries have toDebugString utility function, in order to get the corresponding native JSON representation of the query as a string (useful for debugging purposes)

GH-83

Thanks @anskari. I’ll have a look at them this friday!


Improve ockam reset by display information about what exact has been deleted

Current behavior
When a user runs ockam reset -y or just ockam reset exactly what types of information are deleted isn't shown currently. Here are some screenshots.

Expected behavior
Instead it would be more useful to a user if the types of information being deleted would be listed with a checkbox beside them indicating that their deletion / removal is successful.
Here's an example that  illustrates the issue:

In the above screenshot it shows that there where at least 1 vault, 2 identities and 1 node on the system. But ockam reset doesn't print information about what it actually deletes. The goal of the ticket is to change that so ockam reset prints to stdout all items that it has deleted.
Please take a look at this file to get started.

We love helping new contributors! ❤
If you have questions or need help as you explore, please join us on Discord. If you're looking for other issues to contribute to, please checkout our good first issues.

Hi @nazmulidris , I'd like to work on this as well.

@Abhicodes-crypto That's awesome, this is all yours. Please let us know if you have any questions as you explore. You can also ask questions on the contributors discord https://discord.gg/RAbjRr3kds

@Abhicodes-crypto Continuing our chat from Discord, I am closing this issue.
Thank you for taking a look at this issue 🙏🏽 . Sorry we had to close it. There are many that are still open if you want to take a look 👍🏽
https://github.com/build-trust/ockam/issues?q=is%3Aissue+is%3Aopen+label%3Ahacktoberfest
More info (captured from Discord convo):

@adrianbenavides : This one is a little bit tricky and I'm not sure it has a clear solution right now. The code that handles the state deletion doesn't know anything about stdout/stderr and from the command side we don't want to know the details about how the state is deleted (we just call CliState::delete()). Since the Terminal struct (the object we use to handle stdout/stderr) lives in ockam_command and CliState lives in ockam_api, we can't pass the Terminal object to CliState::delete(), so we are basically in a dead end here.


Add UML directory in docs

Need some diagrams to clarify workflow.

Commit ede6234 adds the data acquisition diagram.

Commit db1a31f v0.0.5, UML directory added to the stable branch.


CasePathable.modify: Change XCTFail to runtimeWarn as per documentation

As per the documentation:

If the enum’s case does not match the given case key path, the mutation will not be applied, and a runtime warning will be logged.

In reality, calling modify with a non-matching case will always call XCTFail. This PR adds ComposableArchitecture's RuntimeWarnings.swift file and uses runtimeWarn instead of XCTFail.
Alternatively, we could consider changing the documentation to clarify that the use of a non-matching case causes a runtime crash. I'm open to either solution, but a warning log seems a bit more graceful.

Hi @djangovanderheijden, our XCTFail from xctest-dynamic-overlay does a runtime warning under the hood when not in tests.
So this does currently behave as described in the docs. Are you seeing otherwise? Also, this definitely should not crash the app. Are you seeing that?

Hey @mbrandonw, interesting. I did see a crash in my application (with the debugger attached, but not while testing). I'll take another look and get back to you

Alright, the issue seems unrelated to CasePaths (sorry about that, I should've read the error on my colleague's screen better) but I am definitely seeing crashes:

This happens in a clean, newly created project that's ran for the first time. Here's a small repro:
@CasePathable
enum Foo {
    case bar(String)
    case baz
}

struct ContentView: View {
    var body: some View {
        Text("Foo")
            .onAppear {
                var foo = Foo.baz
                foo.modify(\.bar) {
                    $0 = "bar"
                }
            }
    }
}

Nice! I'll close this.


is there any way to get selected all data  instead of id? ~Event: 'nodeSelected'

Have you tried exporting?
const data = editor.export();

Yes it is all data, but I need to get only clicked item

Try:
    editor.on("nodeSelected", (id) => {
      const dataNode = editor.getNodeFromId(id);
      console.log(dataNode);
    })

awesome thank you!


/concepts and /concept endpoint seems broken

Requests to /concepts endpoint don't work no more. Raises 500 status code, and doesn't seem to make it to Sentry.
Seems to apply to all concept schemes.
Steps to reproduce the behavior:

Go to https://api.g.sentier.dev/latest/concepts?concept_scheme_iri=http%3A%2F%2Fdata.europa.eu%2Fz1e%2Ficstcom2009%2Ficst&lang=en
Cry

Seems to be the same for /concept, e.g. https://api.g.sentier.dev/latest/concept?lang=en&concept_iri=http%3A%2F%2Fdata.europa.eu%2Fz1e%2Ficstcom2009%2F10.

Cry

I did. And now sentry knows about it

ProgrammingError

(psycopg.errors.UndefinedColumn) column in_scheme_1.member_iri does not exist
LINE 3: ...iri) AS anon_1 ON anon_1.collection_members_iri = in_scheme_...
                                                             ^
[SQL: SELECT concept_schemes.iri AS concept_schemes_iri, concept_schemes.notation AS concept_schemes_notation, concept_schemes."scopeNote" AS "concept_schemes_scopeNote", concept_schemes."prefLabels" AS "concept_schemes_prefLabels", anon_1.collection_members_iri AS anon_1_collection_members_iri, anon_1.collection_members_notation AS anon_1_collection_members_notation, anon_1."collection_members_prefLabels" AS "anon_1_collection_members_prefLabels", anon_1.collection_members_member_type AS anon_1_collection_members_member_type, anon_1.collections_iri AS anon_1_collections_iri, anon_1.concepts_iri AS anon_1_concepts_iri, anon_1.concepts_identifier AS anon_1_concepts_identifier, anon_1."concepts_altLabels" AS "anon_1_concepts_altLabels", anon_1."concepts_scopeNotes" AS "anon_1_concepts_scopeNotes"...

Thanks @sami-m-g. Why did that solve the issue?


Remove user-provided Transaction timestamp

See #808. This includes https://github.com/bigchaindb/bigchaindb/pull/816 also. If this gets merged I'll do the same for bdb cli and the driver.

The Travis CI tests are failing because of two small flake8 errors --- errors which should now be fixed in the master branch (since PR #765 got merged).

Current coverage is 96.78% (diff: 100%)

Merging #817 into master will increase coverage by 0.05%

@@             master       #817   diff @@
==========================================
  Files            26         26
  Lines          1683       1681     -2
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           1628       1627     -1
+ Misses           55         54     -1
  Partials          0          0

Powered by Codecov. Last update 061ec0a...324e877

Thanks, did a rebase.

do we have a way to figure out which block a transaction belongs to?
Otherwise it is very tedious to know at which time a transaction occurred (or at which block_number)
In that case it becomes really hard to use BDB as a timestamping blockchain

@diminator, it seems though that the issue of using BDB as a timestamping blockchain and the utility of the user provided timestamp field are separate issues. Perhaps we should have another ticket for the former?

Perhaps we should have another ticket for the former?

Yes, making this change really needs the other ticket of figuring out which block the transaction belongs to (and maybe additionally providing a block sequence number)

@diminator @libscott Yes, getting all the (node-generated) timestamps associated with a transaction will take a bit of effort:

Get the block the transaction is in
Get the timestamp from that block, somehow (mabye by getting the whole block and parsing it)
Get the votes on that block
Get the timestamps in those votes, somehow (maybe by getting all the contents of those votes and parsing them)

Anyway, those are separate things to do outside the scope of this PR.

do we have a way to figure out which block a transaction belongs to?
Otherwise it is very tedious to know at which time a transaction occurred (or at which block_number)
In that case it becomes really hard to use BDB as a timestamping blockchain

In these cases, would adding a timestamp yourself (perhaps verifiable, e.g. OpenTimestamps) in the metadata not be sufficient? This might also be alleviated if a time oracle were present.

@sbellem It looks like we'll be using the transaction id to identify assets in the future (i.e. the id of the CREATE transaction where the asset was created). The asset.id will be going away. See the discussion on issue #697
This raises an interesting question: if there are two transactions with the same id (hash), then is it okay to delete one (because it's presumed identical)?

This raises an interesting question: if there are two transactions with the same id (hash), then is it okay to delete one (because it's presumed identical)?

Two transactions with the same ID are the same transaction. Currently I'm not sure if we do check for this when we write a transaction to the backlog or to a block.

NOTE: The time stamp (and asset id) play a critical role in the uniqueness of the transaction id. If both are removed two transactions can easily hash to the same value, and consequently have identical ids.

Correct, and #697 relies on proper incentivisation ie of the user who CREATEs an asset to not collide with his/her own transactions. In this case, a signature must be part of the transaction hash.

@libscott The signatures (fulfillment URIs) aren't currently included in what gets hashed to calculate the transaction hash (transaction id). I suppose they could be. Maybe make a new issue for that, including an explanation of why it's necessary (in the new issue's opening comment).

There is one use case where we did trust the user provided timestamp and that was with timelocks.
In this case the user had an incentive to provide a correct timestamp.
When re-adding the timelock functionality to the server the only other way I see is to use the timestamp of the block in which the transaction was included but this approach has two problems:

The client will need to trust the server provided timestamp:
The timestamp will be of when the transaction was validated and not when the transaction was created.

@diminator do you have any input for this since you have more knowledge about timelocks

Today, each node must validate each transaction, usually. Why not leave the determination of the "current time" up to each node: the node would just get the current time as-needed, from its own clock.
If it's definitely before the timeout time, then most nodes will agree that now is before the timeout time. If it's definitely after the timeout time, then most nodes will agree that now is after the timeout time. They'll only disagree, maybe, when now is near the timeout time, and then voting will decide.
I think that approach would be the most decentralized.
For people who care about super-duper time precision, then one option would be to rely on some trusted timestamp from a trusted timestamp provider (i.e. an oracle, a third party). The trusted timestamp could be put in the transaction's metadata. Support for that feature could be added later if there's enough demand.

I understand what you are saying and there is some confusion about how the timelock works at the moment.
When creating a timelock a user did not actually give a time, just an offset in seconds. Something like this timelock expires 30 min after the creation of this transaction.
With what you are proposing we will need to change the actual timelock condition to carry a timestamp instead of an offset

The timelocks work currently by having each node executing now() upon validation of the transaction, and comparing the local node time with a UNIX timestamp (the expiry_time).
This poses a problem when validating a timelocked transaction in the future - after the expiry time.
The user would claim that the transaction time needs to be frozen (timestamped) upon submitting it to BDB. Or at least, the time at which each node in the system sees the transaction.
FYI: In the future-validation can happen in a few scenario's, eg:

when a block was INVALID due to another transaction and the transaction bounces back and forth between the backlog and INVALID blocks.
when you want to validate an entire asset history

One suggestion, made by @libscott is to use a locktime instead of expiry_time. This would mean that a transaction needs to wait a while to become valid - but it would remain valid in all future validations. This becomes more complicated when using execute_condition and abort_condition in hold/escrow transactions.

With what you are proposing we will need to change the actual timelock condition to carry a timestamp instead of an offset

agreed

OK
So since the timelock functionality has not yet been implemented by common now it would be a good time to redesign this feature.
Regarding this pr it is safe to remove the transaction timestamp if we change the timelock condition to carry a timestamp instead of a time offset. This way we can add server side validation code in which each node compares this client provided timestamp with the current server time

@diminator pointed out I was wrong. The timelock already contains a unix timestamp so this pr does not interfere with the timelock conditions

Cool. @sbellem, are we good to go?


Feature Compat: Escape Rooms

https://wide-cream.glitch.me is a good one to fork. Source code here: https://glitch.com/edit/#!/wide-cream
The roomscale in Vive is good. Maybe it's my setup, but in the Rift, the stage is a bit too small to play.
It's nice to see the controllers well work across the Vive and Rift. Note: The collision bounding boxes should be a bit larger for the aframe-super-hands component (which is what's used for the universal controls under the hood).
@caseyyee: This would be a good project to start as a "conformance test." (Which reminds me we should have something like http://acid3.acidtests.org/ for WebVR.)


Support server-side rendering

There's a lot of benefits to being able to render components server-side. Would be amazing if uix supported this.

Tradeoffs in server side and client side rendering
I really don't want to create layers of leaky abstractions just to support content publishing websites that actually doesn't have complicated user interfaces on pages that display content.
Right now I just don't see any good way to implement API for building complex user interfaces in Dart that will work on client and server.

I really don't want to create layers of leaky abstractions just to support content publishing websites that actually doesn't have complicated user interfaces on pages that display content.

Any website can benefit from an isomorphic approach, not just content publishing sites. Montage is one where we'd like to take advantage of server-side rendering. Even intranet sites mentioned in the article you listed would benefit from this approach. Keep in mind that intranet users are users too, and making the perceived performance of a site faster benefits the user's experience.

Right now I just don't see any good way to implement API for building complex user interfaces in Dart that will work on client and server.

Which API are you referring to, uix?

Which API are you referring to, uix?

any API that doesn't hide dart:html with some really bad abstraction.
When configured imports will be implemented in Dart, I'll add support for server-side rendering, it is actually quite easy to do.
I just don't want to "fix the web" with some broken abstraction on top of dart:html that will work only in simple cases. If Dart adds support for conditional compilation it will be significantly easier to write "isomorphic" components with uix, all other ways that I can think of will fail as soon as component requires non-trivial form of interaction.

I'm not sure if configured imports are the right answer to this problem, as it doesn't abstract the components from the environment for the developer.
With a configured imports approach, it places more burden on the developer to wrap any code that uses dart:html with conditional blocks, i.e. // @ifdef BROWSER and // @endif. This'd become especially tedious when working with browser events.
For abstracting events, I think introducing a concept of synthetic events is the right place to start. It could also serve as the basis for dealing with cross device events, if you ever wanted uix to work on -- fingers crossed -- Dart mobile.

There is no point to abstracting environment from the developer, it is quite important to understand that server-side rendered html output should not always be equal to progressive enhanced html to provide good user experience. For example, Flickr is an example of a good app that can render html on the server and progressively enhance when js is loaded, even when js isn't fully loaded and user tries to interact with some non-trivial input element, they're providing some form of feedback with a small js code that is injected in every html. Implementing a good "isomorphic" app will require to write specific code for browser and server environments.
uix will never work on native mobile platforms like React native, it is not that hard to implement similar project that will work on mobile platforms, and provide similar experience for building UIs. I just don't believe in "one size fits all" solutions, native mobile platforms have different set of problems.
Even in React if you try to implement proper material design Button, it will require access to the native dom api.


build!: upgrade engines field to >=8.10.0

This drops support for all Node.js versions < 8.10.0, please upgrade
BREAKING CHANGE: library now requires Node >= 8.10.0

This should wait until I have made another release on the 3.x branch (pending)


Config: now that dev_type is supported, use this in client_generate_connect_options as well

missing from #12

to sum up, as discussed: Conf_map gets two keys Dev : string k and Dev_type : [ `Tun | `Tap ] k.
a configuration entry dev tun100 should insert both Dev "tun100" and Dev_type `Tun into the configuration map (unless dev-type is already present). Whether to fail on
dev-type tap
dev tun123

is not clear, if not the Conf_map should contain Dev tun123 and Dev_type `Tap (i.e. do not overwrite with  `Tun).

superseeded by #17


How to forceupdate(redraw) a listView?

With new data? Or with the same data? If with the same data, why do you need to redraw it?
Help us help you - we need more detail here!

I change the this.state.currentCatindex in other place, but the below listView's renderRow not redraw.
renderSider: function(item) {
    var me = this;
    var viewStyle = [styles.catItem];
    var textStyle = [styles.catItemText];

    if(item._index == this.state.currentCatIndex) {
      viewStyle.push(styles.currentCatItem);
      textStyle.push(styles.currentCatItemText)
    }
    return (
        <View style={viewStyle} onTouchStart={(event) => me.handleCatItemClick.call(me,item._index)} >
          <Text style={textStyle}>
            {item.title}
          </Text>
        </View>
    );
  },

Please use Stack Overflow for questions: http://stackoverflow.com/questions/tagged/react-native
Post your question with the code sample there and close this issue.

+1 @KJlmfe did you find a solution for this? struggling with the same issue

Take a look at your rowHasChanged function on your DataSource
That said, closing because @mkonicek is spot on :smile:

@mkonicek What would happen if we could reference a component inside the render function? Does this go against react principles? For example to add a reloadData() to the listView component.


[#2781] pinpoint 1.6.1 release

version change (1.6.1-SNAPSHOT -> 1.6.1)

Codecov Report

Merging #2782 into master will increase coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2782      +/-   ##
==========================================
+ Coverage    40.7%   40.74%   +0.04%
==========================================
  Files        1773     1773
  Lines       61115    61115
  Branches     8586     8586
==========================================
+ Hits        24876    24902      +26
+ Misses      36239    36213      -26

Impacted Files
Coverage Δ

...m/navercorp/pinpoint/test/OrderedSpanRecorder.java
77.9% <0%> (ø)
:arrow_up:

.../profiler/sender/StandbySpanStreamDataStorage.java
81.35% <0%> (+1.69%)
:arrow_up:

.../pinpoint/rpc/client/PinpointClientHandshaker.java
74.3% <0%> (+6.94%)
:arrow_up:

...rp/pinpoint/rpc/control/ControlMessageDecoder.java
80.35% <0%> (+14.28%)
:arrow_up:

...a/com/navercorp/pinpoint/rpc/util/StringUtils.java
28.57% <0%> (+28.57%)
:arrow_up:

.../java/com/navercorp/pinpoint/rpc/cluster/Role.java
71.42% <0%> (+71.42%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe79ba8...e99a780. Read the comment docs.


How can feature flags in the provenance better reflect the execution of the pipeline run?

Considering the use case of a pipelineRun for this issue (same applies to the taskrun as well).
Feature-flags are used by the validation web hook and the reconciler to guard the API spec usage. We currently store them in the PipelineRun.Status.Provenance field. The feature-flags are used by Tekton Chains and stored in the generated provenance to indicate that the pipelinerun (or the build) was executed with these feature-flags. However, the feature-flags can be updated on the fly by updating the configMap. If this happens mid-way during the build process, the information surfaced in the provenance is incomplete because the flags were mutated during the pipeline run. This means that some pipeline tasks could have run with a different set of feature flags.
Proposed solutions in Issue https://github.com/tektoncd/pipeline/issues/5999 suggested the use of the status field instead of context.
The challenge was as follows:

Validation happens in two places: webhook and reconciler.
If the feature flags are saved in the status of the taskrun/pipelinerun then when we validate a stand-alone pipelineSpec/taskSpec (particularly at the webhook stage), this taskrun/pipelinerun status field is not accessible and so we cannot rely on it.  The status idea would work if we were only validating at the reconciler level.
This means that we need to rely on the context. Now, the challenge here is that the webhook and controller watches for updates to the config map and the context is updated when changes are detected (I think. Please correct me if I'm wrong). This means that we cannot rely on the context directly as well.

This issue is to brainstorm about how the feature flags in the provenance can better reflect the execution of the pipeline run.

Thanks Chitrang! Just want to note one additional comment you made: "The feature flags in the provenance... assumes that both pipeline and pipelinerun were validated with the same set of feature flags".
Currently, Pipelines are validated on creation and again when they are used in PipelineRuns. Feature flags might have changed since pipeline creation, but the value of feature flags when the pipelinerun is executed should reflect what was used to validate the pipeline spec. The one exception is "enable-api-fields" as of https://github.com/tektoncd/pipeline/pull/6701.

Execution of the pipeline/task spec.
eg. after something like kubectl apply -f pipeline.yaml.

the validation web hook validates the API and stored it into etcd. Note that at this point, the feature flags stored in the config map are used during validation.

Execution of the pipelinerun spec.
eg. after something like kubectl apply -f pipelinerun.yaml.

Note here that the feature flags are used in multiple places when reconciling a pipeline run. A change at any point here could result in a new set of feature flags being used for partial build.

The idea of a mutating admission webhook seems reasonable to me.
Instead of confirming that feature flags haven't changed, would it make sense to just use the feature flags from the status rather than feature flags from the configmap? If we threw an error if feature flags changed, this would make it hard for cluster operators to change the value of feature flags without causing running pipelineruns to fail.
It would be interesting to see if we can pass feature flags to the taskrun from the pipelinerun in some way-- I'm not sure if you can create a taskrun with a status.

If this happens mid-way during the build process, the information surfaced in the provenance is incomplete because the flags were mutated during the pipeline run. This means that some pipeline tasks could have run with a different set of feature flags.

This is indeed true, but is it something we want to disallow or not ? A bit similar to the "spec" of a Pipeline being stored in the PipelineRun.status to make sure if it changes we are not affected, it seems to make sense.

Proposed solutions in Issue #5999 suggested the use of the status field instead of context.

What does context mean here ? (is it context.Context in code ? if so, how/why do we compare a field from something  in code that is purely an implementation detail ?).
In code we can use context even if we change how the content of the context is populated.

We could use the admissions mutating webhook to capture the feature flags for the pipeline run in its status.

This is what the "defaulting" admission controller does today yes.

6. We use the owner references for the task and compare the feature flags stored in the taskrun's status to that in its owner (ie. pipelineRun's status). This would ensure that the task run was running with the same feature flags as its owner.

This would be very slippery to do, as essentially, this would mean the TaskRun controller needs to query PipelineRun, making a kind-of cyclic loop between the 2 controllers. Up until now, the TaskRun never has to know where it comes from (PipelineRun, standalone, something else), and it's for the best — e.g. if I have my object called Build and I set it as ownerref of the TaskRun, how can the TaskRun controller verify anything from it if it doesn't know it ? I have a suggestion for this later on.

Instead of confirming that feature flags haven't changed, would it make sense to just use the feature flags from the status rather than feature flags from the configmap? If we threw an error if feature flags changed, this would make it hard for cluster operators to change the value of feature flags without causing running pipelineruns to fail.

Definitely yes.

It would be interesting to see if we can pass feature flags to the taskrun from the pipelinerun in some way-- I'm not sure if you can create a taskrun with a status.

It shouldn't be possible to create an object with an already populated status nope. Even to mutate status, through kubectl for example, you need to use the --subresource='status' (in kubectl patch) to actually make it work.
I think the "worry" here is that we could create a Pipeline that is valid with a set of feature-flag and not with another one, right ? Making the execution kind-of hellish, because you could end-up with a Pipeline in your cluster that is invalid according to the "current" values of the feature-flag configuration map.
This is the behaviour today, and there is little we can do about it really. I do think we should adopt the simplest path (both for users and implementation). We need to decouple (in our head) the Pipeline and the PipelineRun and where the Pipeline comes from — what I mean here is, from the PipelineRun perspective, there should be absolutely no difference if the referred Pipeline comes from in-cluster or outside (embedded, resolvers, …).

The Pipeline referred by the PipelineRun is invalid with the feature-flags at the time of creation, the reconciler fails the PipelineRun directly (with an error that is very understandable for the user)
Same with Task and TaskRun
Changing a Pipeline that is in cluster but with invalid spec with the feature-flags at the time of mutation, will be validated with the current feature-flags (it could surprise the user but.. meh).
Finally, one suggestion here would be, to think of a way to pass the feature flags per TaskRun (and probably even per object) and per-namespace as well (see TEP-0085)

With the proposed suggestion above, the existing defaulting admission controller could set those "feature flag" value per object (let's say, using annotations), and the controllers (PipelineRun and TaskRun) would rely on those. And if not present, rely on the configmap, as it works today.

Thanks for your feedback @vdemeester!!

If this happens mid-way during the build process, the information surfaced in the provenance is incomplete because the flags were mutated during the pipeline run. This means that some pipeline tasks could have run with a different set of feature flags.

Proposed solutions in Issue #5999 suggested the use of the status field instead of context.

What does context mean here ? (is it context.Context in code ? if so, how/why do we compare a field from something in code that is purely an implementation detail ?).

Yes, I meant context.Context which in turn gets it from the config map that is watched by the controllers.

We use the owner references for the task and compare the feature flags stored in the taskrun's status to that in its owner (ie. pipelineRun's status). This would ensure that the task run was running with the same feature flags as its owner.

This would be very slippery to do, as essentially, this would mean the TaskRun controller needs to query PipelineRun, making a kind-of cyclic loop between the 2 controllers. Up until now, the TaskRun never has to know where it comes from (PipelineRun, standalone, something else), and it's for the best — e.g. if I have my object called Build that creates a TaskRun and thus I set it as ownerref of the TaskRun, how can the TaskRun controller verify anything from it if it doesn't know it ? I have a suggestion for this later on.

Yes, I also feared this case also from the side of performance (ie. querying the parent object every reconcile loop).

It would be interesting to see if we can pass feature flags to the taskrun from the pipelinerun in some way-- I'm not sure if you can create a taskrun with a status.

It shouldn't be possible to create an object with an already populated status nope. Even to mutate status, through kubectl for example, you need to use the --subresource='status' (in kubectl patch) to actually make it work.

I agree, I tried creating a taskrun with a status but it did not work. The status of the created taskrun did not have what I had populated.

I think the "worry" here is that we could create a Pipeline that is valid with a set of feature-flag and not with another one, right ? Making the execution kind-of hellish, because you could end-up with a Pipeline in your cluster that is invalid according to the "current" values of the feature-flag configuration map.

I think the worry here is more, mutating the feature flags in the middle of the pipeline run. Basically, we cannot claim that the feature flags in the provenance produced the build.
If we say that the build starts from the creation of the pipeline run then like you say below, it does not matter what the feature flags were when the pipeline spec was applied because the validation will make it fail with a clear message.

This is the behaviour today, and there is little we can do about it really. I do think we should adopt the simplest path (both for users and implementation). We need to decouple (in our head) the Pipeline and the PipelineRun and where the Pipeline comes from — what I mean here is, from the PipelineRun perspective, there should be absolutely no difference if the referred Pipeline comes from in-cluster or outside (embedded, resolvers, …).

The Pipeline referred by the PipelineRun is invalid with the feature-flags at the time of creation, the reconciler fails the PipelineRun directly (with an error that is very understandable for the user)
The Pipeline referred by the PipelineRun is valid with the feature-flags at the time of creation, the reconciler (or the defaulting admission controller) stores the feature flags value in the PipelineRun (somewher) and relies on these for the rest of the execution.
Same with Task and TaskRun for the 2 points above.
Changing a Pipeline that is in cluster but with invalid spec with the feature-flags at the time of mutation, will be validated with the current feature-flags (it could surprise the user but.. meh).
Finally, one suggestion here would be, to think of a way to pass the feature flags per TaskRun (and probably even per object) and per-namespace as well (see TEP-0085)

With the proposed suggestion above, the existing defaulting admission controller could set those "feature flag" value per object (let's say, using annotations), and the controllers (PipelineRun and TaskRun) would rely on those. And if not present, rely on the configmap, as it works today.

I did think of using annotations earlier but the challenge is the the annotations can be modified. If we embed these in the status, then we can use spire integration to guarantee that the feature flags were not modified in the middle of the run.

Also one thing to note : we do not expect users to change feature-flags on a regular basis 👼🏼
Agreed.

Another possible solution:
What if we create a new config map/secret per pipelinerun that stores a snapshot of the feature flags from the moment that the pipelinerun object is created. The key could be the pipelinerun meta UUID which allows the feature flags for the pipeline and its underlying tasks to be fetch from cleanly. Once the pipelinerun finishes, it deletes this resource.
This way, only the operators with privilege to update the config maps can actually modify it in the middle of the run. The end users cannot fake them into the provenance.

/remove-lifecycle rotten


[ASK] Retrieving latent factors from VW matrix factorization

Description

I'm using an Azure DSVM that has VW pre-installed and running the VW deep dive notebook. It shows 2 matrix factorization recommender methods and how to do training and scoring from them, but I would like to retrieve the latent factors or decomposed matrices from the methods for further processing in python. Or at least output the readable models or weights to a text file.
I cannot find/run the gd_mf_weights file described from the VW example (https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Matrix-factorization-example) to produce the latent factors. How can I get the latent factors from the --rank and --lrq models?
Other Comments
It is a great deep dive tutorial overall, but I just have this specific ask. Please advise. Thank you!

Right, currently on DSVMs only the VW binary exists. You can however use the invert_hash option to generate a more readable model.
example:
vw --lrq ui7 -d train.dat -f vw.model --invert_hash vw.model.txt
which will give you a file that contains the item, user, and global biases, as well as low rank weights for each item and user seen in train.dat
example, the weight vector for item 1058 is [-0.425035, 0.042227, 0.293272, 0.161824, -0.286626]
lrq^i^1058^1:104705:-0.425035
lrq^i^1058^2:104706:0.042227
lrq^i^1058^3:104707:0.293272
lrq^i^1058^4:104708:0.161824
lrq^i^1058^5:104709:-0.286626

doing the same for --rank models produces something a little harder to parse, you would need figure out what the columns are there.. =/
it is possible to clone the repo on DSVM and build vw + library files to use gd_mf_weights. Note you will need to retrain your model using the new vw binary otherwise the model files will not be compatible. (or you could checkout the 8.1.1 release version and build that)

If I have the Ubuntu DSVM, does it already have the necessary tools to build vw + library files once I download the 8.1.1 release version there? (Meaning I can open up the terminal and run the build/install commands as described in VW wiki?)
Thank you so much for your help!

yes, you can do that on an ubuntu dsvm
git clone https://github.com/VowpalWabbit/vowpal_wabbit.git
cd vowpal_wabbit
git checkout 3cf3f69279eab5a8f580c58b8eb5a3c87dd984df
make
library/gd_mf_weights --help

Let us know if you have further questions or issues, closing this for now.

I trained a rank 40 model on the movielens data, but cannot retrieve the weights from the trained model with gd_mf_weights. I'm following the syntax from the matrix factorization example but it is giving me errors. Any advice?
Model training call:
vw --rank 40 -q ui --l2 0.1 --learning_rate 0.015 --decay_learning_rate 0.97 --power_t 0 --passes 50 --cache_file movielens.cache -f movielens.reg -d train.vw
Weights generating call:
library/gd_mf_weights -I train.vw -O '/data/home/mlteam/notebooks/Recommenders-master/notebooks/Outputs/movielens' --vwparams '-q ui --rank 40 -i movielens.reg'
Error:
WARNING: model file has set of {-q, --cubic, --interactions} settings stored, but they'll be OVERRIDEN by set of {-q, --cubic, --interactions} settings from command line.
creating quadratic features for pairs: ui

finished run
number of examples = 0
weighted example sum = 0
weighted label sum = 0
average loss = -nan
total feature number = 0
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorboost::program_options::multiple_occurrences >'
what(): option '--rank' cannot be specified more than once
Aborted (core dumped)

If I just run it without specifying rank and interaction variables, it doesn't return the same trained model, since the parameters displayed are different from before.
library/gd_mf_weights -I train.vw -O '/data/home/mlteam/notebooks/Recommenders-master/notebooks/Outputs/movielens' --vwparams '-i movielens.reg'
creating quadratic features for pairs: ui
Num weight bits = 18
learning rate = 10
initial_t = 1
power_t = 0.5
using no cache
Reading datafile =
num sources = 0
Segmentation fault (core dumped)

If I run weights generation with the entire set of model training parameters, it just ignores my extra parameters (and finishes much faster than 50 passes would take) and returns same weights from a randomly initiated rank 40 model.
library/gd_mf_weights -I train.vw -0 '/data/home/mlteam/notebooks/Recommenders-master/notebooks/Outputs/movielens' --vwparams '--rank 40 -q ui --l2 0.1 --learning_rate 0.015 --decay_learning_rate 0.97 --power_t 0 --passes 50 --cache_file movielens.cache -f movielens.reg -d train.vw'

sorry to take so long on this.
you need to be careful with -i, -f when specifying the model when using gd_mf_weights. Also I believe the existing example on vw is no longer correct since the rank and interaction terms are stored in the model. Resulting in the errors you're seeing. My understanding is you should drop the --rank and -q params, but keep the rest (and make sure you reference the model using -i).
Example:
vw --rank 40 -q ui --l2 0.2 --learning_rate 0.015 --decay_learning_rate 0.97 --power_t 0 --passes 50 --cache_file movielens.cache -f movielens.reg -d train.vw
library/gd_mf_weights -I train.vw -O ./results --vwparams '--l2 0.1 --learning_rate 0.015 --decay_learning_rate 0.97 --power_t 0 --passes 50 --cache_file movielens.cache -i movielens.reg'

If there are additional issue, it might be better to raise them with the vowpal wabbit repo: https://github.com/VowpalWabbit/vowpal_wabbit

Thanks, Scott. I did post on VW github and found that the issues had to do with versioning and bugs. https://github.com/VowpalWabbit/vowpal_wabbit/issues/1851
https://github.com/VowpalWabbit/vowpal_wabbit/issues/1871


Could you release the complete code？

Hi，
I encountered difficulties in reproducing the CPO and have to turn to your code.

I forgot to answer this question for a long time. Please refer to the following repo, which contains implementations of PICCOLO, CPO, and the newer work LDL: https://github.com/82magnolia/panoramic-localization.


StorageUse leads to multiple entries for one event

I have found that having "isStorageUseDisabled" set to false will cause some events to be sent multiple times. Well I am not actually sure that they get sent multiple times, but they appear multiple times in the protocols on the azure-portal.
Setting "isStorageUseDisabled" to true resolved the issue for me.
Here is my according question on SO:
https://stackoverflow.com/questions/54929563/azure-application-insights-shows-the-same-event-at-identical-timestamps-multiple/54948285#54948285
I have found another issue here: https://github.com/Microsoft/ApplicationInsights-JS/issues/436 This might actually be the same thing?
Question: When are the events removed from the storage? Only after a call to the tracking-server was successfully completed? I observed that when I reload the page while the tracking-request is running, the events are not removed from the storage. This could explain why sometimes events are duplicated?
(I am using version 1.0.20)

@tobias74  - Items are removed from storage from after successful send. Issue can occur sometimes if you have multiple tabs open. Do you have multiple tabs open?
<EMAIL_ADDRESS>Beta SDK has the ability to provide prefix for localstorage/cookie name. Setting:
{
instrumentationKey: "xxxxxxx"
namePrefix: "App1"
}
If you provide this setting, local storage will be unique between multiple sessions and not cause any duplicates.

Please reopen if you still see issues in beta SDK using namePrefix setting or if multiple tabs are not opened.

For me the issue occurred in case the answer from the insights-server took too long. In my case the answer to the tracking-requests took about 10 seconds to complete. The server had already recorded the events. But when the user navigated (or unloaded) the page while the request to server was still running, the stored events would not be marked as sent, since the http-request would not have had a response yet.
For me this is resolved since I do not use the storage at the time.
Thanks! :-)

Hi @tobias74, I have been trying with the latest version for the library (2.5.4), and still facing this issue if i open multiple tabs. The only way i can avoid duplicated events is by:

Setting "isStorageUseDisabled" to true.
Assigning a namePrefix (unique) for each new session (or tab) i open for the application.

When you referred to the new version (marking the events when the request is sent) which one are you talking about? 2.0.0?
Thanks!!


Source env files if they exist

A local.sh file is supplied to populate environment variables so tests can easily be run locally. When the tests are run via jenkins, these environment variables are populated automatically.
This will source a local.sh if it exists and skips that step if running against preview/staging/prod.

Looks good, I think. 👍


Update presentation images

Hello buttercup team,
I thought to update the presentation images as they have become a bit outdated.
I have update the mobile devices mockups to iPhone 14 and used macOS Monterey to make the buttercup desktop screenshots.
Please check 😄

Thank you @ZanMarolt! This is great :)

Thank you for checking 👍


[gcd-ct-fix] Make the Travis change permanent

Ping @cpereida for review!

Merged! Thanks all :+1:


fixed the bugs of To do component's animated card

closes #140
What was the Issue

when the user was clicking on the checkbox multiple times it was crashing,
when the user was swiping the animated component continuously then also the app was crashing

fixed both the issues
Anyone reviewing this PR, please run it on your phone to check if it is working fine
video for reference
https://user-images.githubusercontent.com/90315175/192683701-797eab74-dbf2-4208-be90-c9b90245df1a.mp4

LGTM


No Support for Android Platform

How would it work? The X11 version uses a python script, but Cpython isn't natively supported on Android. Except Godot can give low level access to kotlin or java api's to run a port of yt-dlp.

With OS.execute + A Custom Android ytdl binary, this feature will be implemented.

Ideas

Modifying a Universial .apk build of Termux for the sole purpose of running yt-dlp commands and supplying this apk for download via github.

Hey :)  If you're motivated to do it go ahead, as long as you keep the same GDScript API.
I really don't have the time for this right now as I have a lot of work for college...


Upgrade to pydantic 2

Hi, we are using pydantic 2 in our orchestration but noticed that datahub requires pydantic <2. Is there a plan to upgrade to pydantic 2? When can we expect it? Downgrading currently is not an option for us.

dagster._core.errors.DagsterImportError: Encountered ImportError: cannot import name 'model_validator' from 'pydantic' (/opt/conda/lib/python3.10/site-packages/pydantic/init.cpython-310-x86_64-linux-gnu.so)while importing module active_metadata_hub. Local modules were resolved using the working directory/opt/dagster/dagster_home/src/active_metadata_hub. If another working directory should be used, please explicitly specify the appropriate path using the -dor--working-directoryfor CLI based targets or theworking_directory configuration option for workspace targets.
This is the error that is caused by pydantic 2.

I have the same problem.
Is that in the roadmap?

Take a look at the note we added to our docs here https://github.com/datahub-project/datahub/pull/11423


Error while parsing

Description
Paket version 4.5.0
Paket failed with:
        Cannot load C:\Users\CameronTaggart\tachyus\thermion\packages\JWT\JWT.nuspec
Message: Error while parsing *
Message: Can't parse "*".
Input string was not in a correct format.

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>JWT</id>
    <version>2.3.1</version>
    <authors>John Sheehan, Michael Lehenbauer, Alexander Batishchev</authors>
    <owners>johnsheehan, devinrader, abatishchev</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <licenseUrl>https://creativecommons.org/publicdomain/zero/1.0/</licenseUrl>
    <projectUrl>https://github.com/jwt-dotnet/jwt</projectUrl>
    <description>Jwt.Net, a JWT (JSON Web Token) implementation for .NET</description>
    <copyright>Public Domain</copyright>
    <language>en-US</language>
    <tags>jwt json</tags>
    <dependencies>
      <dependency id="Newtonsoft.Json" version="*" />
    </dependencies>
  </metadata>
</package>

Repro steps
Please provide the steps required to reproduce the problem
nuget JWT 2.3.1

If possible then please create a git repository with a repro sample or attach a zip to the issue.
Expected behavior
I'm not sure if this is a nuspec authoring error or a paket error.
Actual behavior
Known workarounds
nuget JWT 1.3.4

I think they changed something to allow this nightmare. A fix is underway - but will take some time since I want to run all integration tests

NOTE: Someone should do it properly - I just fixed the error. but it seems nuget ranges are now changed and our parser needs adjustments

also related: https://github.com/fsprojects/Paket/commit/1575266f56b62f155fe8b3cdddc177fe08c7b2a3

reopened in order to let someone else fix it properly.  but paket 4.5.1 release build is running and reported error will be fixed

ok fix is released.

Reading the nuspec/nuget/dotnet docs, I can't find any references to using a wildcard without preceding numbers.
Also, JWT v2.3.1 seems to be no longer available on NuGet, and the next version 2.3.2 does have a version specified for the dependency.
However, all previous beta and alpha versions - from 2.0.0-beta2 up to 2.3.0-beta - still have the bogus dependency, which means Nuget.org does technically allow it.
In version 2.0.0-beta2, where the offending nuspec line first appeared, it had no "version" attribute at all, rather than an asterisk. This means that the asterisk was added during packing. Considering the lack of documentation for this "feature", I'm inclined to think that this is just a bug in dotnet pack.

I think * should be parsed as >= 0

@forki:

But your fix in https://github.com/fsprojects/Paket/commit/3b8047e3c8112a7e0d63ace257f7c98073a25feb already does that, no?

Parsing * as >=0 is inconsistent with dotnet restore. When the spec is ">= 0", Paket will select the highest version, but dotnet restore will select the lowest (I have verified this empirically, just in case). This will cause different resolution of Newtonsoft.Json (and thus potentially different behavior) between Paket-based apps that reference JWT v2.3.0-beta and similar apps that do not use Paket. Is that ok?

Paket had always been different in install behavior. I could write endless essays about that. When we do convert we try to do a conservative conversion and keep it as aligned as possible to nuget. But sometimes that's not really possible.

Ok, I suspected that would be the answer.
But what about the first point? Didn't your commit already make it parse as ">= 0"? Should this case be closed then?

yes I think I did that


[$1000] Web - Chat -If you click on a chat message, press Esc-message is highlighted with a blue frame

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Issue found when executing PR https://github.com/Expensify/App/pull/15650
Action Performed:

Go to URL https://staging.new.expensify.com/
Login with any account
Go to any chat
Write some text
Click outside the compose box - on a message in a chat
Press Esc

Expected Result:
Nothing should happen
Actual Result:
If you click on a message in the chat, press Esc - message is highlighted with a blue frame.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] Windows / Chrome
[ ] MacOS / Desktop

Version Number: <IP_ADDRESS>
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://user-images.githubusercontent.com/93399543/223551176-6a2f2a88-fcdb-4653-90cd-183f03b8c31b.mp4
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~014f93d9e7848a9dc1
Upwork Job ID:<PHONE_NUMBER>979748864
Last Price Increase: 2023-03-08

Triggered auto assignment to @lschurr (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

Bug0 Triage Checklist (Main S/O)

[ ] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
[ ] This bug is not a duplicate report (check E/App issues and #expensify-bugs)

If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead

[ ] This bug is reproducible using the reproduction steps in the OP. S/O

If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
If the reproduction steps aren't clear and you determine the correct steps, please update the OP.

[ ] This issue is filled out as thoroughly and clearly as possible

Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.

[ ] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

I reproduced this in Chrome.

Triggered auto assignment to @pecanoro (Engineering), see https://stackoverflow.com/c/expensify/questions/4319 for more details.

Job added to Upwork: https://www.upwork.com/jobs/~014f93d9e7848a9dc1

Current assignee @lschurr is eligible for the External assigner, not assigning anyone new.

Triggered auto assignment to Contributor-plus team member for initial proposal review - @sobitneupane (External)

I think this issue is related to keyboard navigation or accessibility and they are supposed to be on hold, see the original discussion thread https://expensify.slack.com/archives/C01GTK53T8Q/p1668604822482759

Looks like something related to react-navigation may have been mentioned in this issue discussion.
As a reminder, please make sure that all proposals are not workarounds and that any and all attempt to fix the issue holistically have been made before proceeding with a solution. Proposals to change our DeprecatedCustomActions.js files should not be accepted.
Feel free to drop a note in #expensify-open-source with any questions.

It's how the native elements works. Ref: https://www.github.com/adobe/react-spectrum/issues/2821 (you can test the fiddle on the issue)
So, I don't think this is a bug.

Proposal
Please re-state the problem that we are trying to solve in this issue.
On web, focused elements are displayed with a blue border when the mouse is not over it.
You don't have to do all the steps listed to reproduce this.
All you need to do is, click on any message in any report, press Esc and then move the mouse away from the message.
https://user-images.githubusercontent.com/37259346/223945021-35c6799f-477a-466f-b323-8b363ff16e55.mov
What is the root cause of that problem?
Pressing Esc focuses the message, which was selected by clicking the mouse.
When mouse is over a focused message, the app is programmed to display the context menu.
If the mouse is not over, then :focus-visible CSS styling takes effect, which put the blue border aroud the message due to:

:focus-visible {
    outline: 0;
    box-shadow: inset 0px 0px 0px 1px #5AB0FF;
}

What changes do you think we should make in order to solve the problem?
Prevent the Esc key from focusing the selected message.
We have to override the default behaviour in this case.
I don't have a definite plan to achieve this, but putting this proposal out there just in case it helps someone else find the solution.
Also, I want to stay on the loop with this one.
What alternative solutions did you explore?
Removing the box-shadow removes the blue border from being displayed around a focused message when the mouse is not over it.

:focus-visible {
    outline: 0;
}

But it is better not to change this, as not showing the blue border around focused objects affects the tab key shortcut also.
https://user-images.githubusercontent.com/37259346/223945152-0fb5e725-b8cd-49c0-92c1-79a624b77f64.mov

The issue is because of default browser behaviour, it is a normal behaviour which can be fixed.
To fix the issue I will add a css rule to remove this behaviour and after that it shouldn't appear again, I have managed to remove the issue on my web developer tool as I will show the issue being elemenated in the screen recoding attached.
https://user-images.githubusercontent.com/74202751/223958561-159a238f-6ef3-4ada-bbd7-a8774ea0dc8e.mp4

📣 @AmjedNazzal! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:

Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.

Format:
Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

Contributor details
Your Expensify account email<EMAIL_ADDRESS>Upwork Profile Link: https://www.upwork.com/freelancers/~0113f371085fef1cd8

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

@sobitneupane could you review these proposals?

Proposal
Please re-state the problem that we are trying to solve in this issue.
Clicking a message then pressing the ESC key shows a blue frame.
What is the root cause of that problem?
The ReportActionItem is focusable.
What changes do you think we should make in order to solve the problem?
Negate the focus of ReportActionItem i.e. add onFocus callback here where we blur the element immediately.
What alternative solutions did you explore? (Optional)

Make ReportActionItem un-focusable. This will require RNW changes.

@sobitneupane are you able to take a look at these proposals?

It looks like a valid issue to me. The focused message is highlighted on pressing any key which is not the expected behavior. I believe the expected behavior here is: the focused message should be highlighted only when pressing "Tab" key. And It looks like the it is present throughout the app.
cc: @pecanoro

We are expecting an updated proposals.
cc: @osofus @moezh @mdneyazahmad @s77rt

@sobitneupane my proposal is aligned with the requirements.

I was testing on other apps (Slack) and websites and you are right, no elements besides the ones that we can interact with are focusable. Is there any reason why these were focusable in the first place?

@pecanoro I'm not really sure but maybe it was unintended since the message item is a Pressable it's focusable by default.

@sobitneupane What are your thought, do you think we can remove the focusable property of it without causing any issues?

I agree with @s77rt. My first thought was if we remove focus from ReportActionItem IOU Message and Attachments might not be focusable. But turns out it is not the case. Even when ReportActionItem is not focusable, individual pressable IOU Messages and Attachments will remain focusable.

@mdneyazahmad Thanks for the proposal.
The ReportActionItem will remain focusable and will be focused by pressing Tab with your proposal. We are looking to completely get rid of focusable nature of ReportActionItem as it is not pressable (i.e, no action on pressing)

@s77rt Thanks for the proposal.
Can you please elaborate your proposal.

Make ReportActionItem un-focusable i.e pass focusable={false}.

Can you please add an permalink where you are suggesting this change?

This ^ will require RNW changes here.

Do we need to make any change in react native web? What sort of changes are required?
Please update your proposal and let me know.

@sobitneupane slack also focuses on message list item, but the navigation there is different than ours. When you press up arrow or down arrow, it focusses the corresponding message, in our case, it scrolls.
Selecting prev and next message with arrow is good for accessibility, otherwise it might need multiple Tab to skip messages. When the message list item gets focused, screen reader will read it.
https://user-images.githubusercontent.com/77761491/224937589-52e7b2fd-a5e0-453c-80a0-ff45ebbf5282.mp4

@sobitneupane

The permlink where to add focusable={false}.
RNW had a lot of changes regarding when to apply tabIndex. The easiest way (and the safest) is just to remove that attribute on render as this won't require any RNW changes.

Checking @mdneyazahmad uploaded video, I assume that we will implement arrow navigation for messages in the future, so completely removing focus from message items does not look like as a future proof solution. Updating my proposal based on that...

Proposal
Updated

I assume that we will implement arrow navigation for messages in the future, so completely removing focus from message items does not look like as a future proof solution.

@pecanoro Can you please confirm that? Are we planning to move with arrow navigation for messages in the near future?
I think even if we decide to do so, we can undo the changes. But for now I don't see any reason to keep ReportActionItem  focusable.

@sobitneupane can you please review my proposal? This will work for both keyboard and mouse users.

I think even if we decide to do so, we can undo the changes. But for now I don't see any reason to keep ReportActionItem  focusable.

I believe, we should not remove something that is already working, and that we may decide to reintroduce in future.

@mdneyazahmad how did you manage to select the message on Slack? On the mac app, I can't seem to be able to focus it.

@pecanoro

Have the name focused
Press Shift+TAB (navigate back)

That is incredibly odd so you can only focus it if you navigate back but if you navigate forward... That seems like a bug in Slack hahaha However, I tested on WhatsApp and they do let you focus on each message and then go back and down.
I am not sure if we will implement arrow navigation any time soon, I couldn't find any convos about it. Maybe we should open an issue for it since the other platforms seem to have it.
What I noticed is that, in other apps, when you type a letter, it will just write in the composer but in our case, it just focuses the element, which it's pretty odd. Honestly, the more I look into this, the more I think we should do nothing when it comes to the Esc letter and open another issue to actually fix the problem of typing letters, which should just type in the message composer instead.

issue to actually fix the problem of typing letters, which should just type in the message composer instead.

I agree @pecanoro .

Ok, let's close this one. Thank you everyone for your proposals and sorry for having to close this one. I reported the composer problem in #expensify-bugs here so we can fix that instead.


Is there a tutorial on the ANMM model?

hi @levysouza I'm sorry that this project is not actively maintained (with new tutorials). However, since the API is pretty much the same, you can checkout other tutorials together with ANMM parameters.

I am getting this error:
InvalidArgumentError:  indices[0,0] = 3575 is not in [0, 300)
[[node embedding/embedding_lookup (defined at /home/lss9/anaconda3/envs/matchzoo/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_keras_scratch_graph_1494]
Function call stack:
keras_scratch_graph


Increase cc env coverage

Coverage increased (+3.5%) to 52.052% when pulling 5f0248b86dfd45256c60734ac903d1b26d101c17 on jodh-intel:increase-cc-env-coverage into f24c2b04f2f8bb76d879564a0a9398ba7d99c4a2 on clearcontainers:master.

Coverage increased (+3.5%) to 52.052% when pulling 5f0248b86dfd45256c60734ac903d1b26d101c17 on jodh-intel:increase-cc-env-coverage into f24c2b04f2f8bb76d879564a0a9398ba7d99c4a2 on clearcontainers:master.

lgtm


CREATE USER / GRANT not Recognised as Change

Hi again,
this may be intentional, but just in case it's not:
When I add something like this to my schema.sql and run a atlas migrate diff it will not be recognised as change and thus no migration file will be created:
do
$$
    begin
        create role rds_iam;
    exception
        when duplicate_object then raise notice '%, moving to next statement', sqlerrm using errcode = sqlstate;
    end
$$;

create
    user tariffservice with login;
grant rds_iam to fooservice;

create
    user tariffsink with login;
grant rds_iam to barservice;

Not a big deal, as I can create the diff file manually.
Thanks!

Yes, the main reason because atlas doesn't support for create user / role yet. :D


feat(lab-2558): allow to import jp2 and nitf images

⚠ Label import needs to be deployed in prod before merging this MR.


Pass the webpack stats to renderToHtml method so we can extract js chunks into html

The existing Dynamic Import didn't support SSR.  It meant the rendered html was always blank.  Using react-universal-component, we can SSR these dynamic imported chunks :)
I updated the example for "dynamic import" ~ if we want a separate example instead let me know?
Thanks

Looking good. Just double check nothing is getting committed or changed that doesn't need to be.

Also I'm not sure why it's showing changes to the readme. Make sure you pull in any updates from master

yeah something got janky with the readme... I tried to recopy it from master but must have messed up.
Question.  Should this be a new example or updating old dynamic import one?  Might be useful to have and “advanced dynamic import” example

@rileylnapier The existing Option 1 doesn't work with SSR for you?

It should probably be moved to the constructor, since componentDidMount in not run in SSR. Don't you think that would be an easier solution? Again, I might be overlooking something here.
constructor (...args) {
  super(...args)
  if (typeof document === 'undefined') {
    const { MyComponent } = require('../components/MyComponent')
    this.state = { MyComponent };
  } else {
    this.state = { MyComponent: null }
  }
}
// /////////////////////////////////////////////////////////////////////////
// // Option 1
// // Behaviour:
// //   - Loaded component on static build
// //   - UnLoaded component when react rehydrates
// //   - Loaded component when dynamic import resolves
componentDidMount() {
  import('../components/MyComponent')
    .then(({ MyComponent }) => this.setState({ MyComponent }))
    .catch(error => this.setState({ MyComponent: Failed }))
}

dynamic imports don't render anything with SSR.  you can show this by running the existing "dyanmic imports" example, "npm run build && npm run serve" then go to /about page.  it will always show the "loading" first since the page doesn't have any ssr.

Yes, that's because Option 1 is commented out in the starting dynamic imports example. If you change to my previous comment, you will have MyComponent rendered in SSR.

the problem in option 1 is:
UnLoaded component when react rehydrates

if anything... we don't have to include this in the "examples" ~ but i would still require passing of the client/prod webpack stats to the renderToHtml function

Gotcha. Yeah, it's a very powerful addition. Then the user can even use e.g. React Loadable and other alternatives as well.
react-static is very much Tanners baby, but just giving my two cents: Wouldn't it then be cool with a separate code splitting example, and a separate dynamic-imports?

Yeah I think I’ll do another example “advanced dynamic import with ssr”
I originally tried to do this with react-loadable but couldn’t get it to work.  Also that repo has issues turned off so I couldn’t see if anyone else in the past had figured out the issue or ask any questions.  Super odd to me... so I went with react-universal-component instead.

I am perfectly fine with the stat additions. I agree that this should
likely be a different example entirely. I would also be interesting in
discussing the ramifications of sprin supporting this in the core. Is it a
90% use case? Would it hurt users who dont utilize it?
On Sat, Dec 9, 2017 at 6:21 PM Riley<EMAIL_ADDRESS>wrote:

Yeah I think I’ll do another example “advanced dynamic import with ssr”
I originally tried to do this with react-loadable but couldn’t get it to
work. Also that repo has issues turned off so I couldn’t see if anyone else
in the past had figured out the issue or ask any questions. Super odd to
me... so I went with react-universal-component instead.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nozzle/react-static/pull/230#issuecomment-350520269,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFUmCQ72g4jdz7y0Y5ltQ80PEFm-O9Fzks5s-0AygaJpZM4Q8PII
.

Sprin supporting in its core?  Typo?
Do you mean adding this to the core library?  I would support that as the config spans a few different places and because of that it becomes fragile.
Either that or chat about a pluin system?
Also, on a separate note @tanner,  can I schedule some time with you to explain how my company is planning on using the library and chat with you on future plans/roadmap for react-static?

Yeah, it would be cool with some sort of opt-in code splitting HOC in core.
<CodeSplit withRouteProps loading={() => <div> Loading... </div>}> {/* ... */} </CodeSplit>

Agreed. Gatsby does code splitting but it's extremely baked in to the setup
and configuration. If we could allow the same technical support for it
without breaking anything, that would be fantastic. It may not be automatic
like Gatsby, but it would cover that small percentage of sites that may
have massive app bundles. Honestly a lot of the code in that example looks
scary to me right now. Not sure if that's because it's foreign to me or if
it feels fragile.
On Sun, Dec 10, 2017 at 6:14 AM Emil Tholin<EMAIL_ADDRESS>wrote:

Yeah, it would be cool with some sort of opt-in code splitting HOC in core.
<CodeSplit withRouteProps loading={() =>  Loading... }> {/* ... */}
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nozzle/react-static/pull/230#issuecomment-350551181,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFUmCfbhZMPzRlAniSkthluc6GSVof-Jks5s--dKgaJpZM4Q8PII
.

@rileylnapier feel free to jump on the slack org and DM me.
On Sun, Dec 10, 2017 at 6:23 AM Tanner Linsley<EMAIL_ADDRESS>wrote:

Agreed. Gatsby does code splitting but it's extremely baked in to the
setup and configuration. If we could allow the same technical support for
it without breaking anything, that would be fantastic. It may not be
automatic like Gatsby, but it would cover that small percentage of sites
that may have massive app bundles. Honestly a lot of the code in that
example looks scary to me right now. Not sure if that's because it's
foreign to me or if it feels fragile.
On Sun, Dec 10, 2017 at 6:14 AM Emil Tholin<EMAIL_ADDRESS>wrote:

Yeah, it would be cool with some sort of opt-in code splitting HOC in
core.
<CodeSplit withRouteProps loading={() =>  Loading... }> {/* ... */}
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nozzle/react-static/pull/230#issuecomment-350551181,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFUmCfbhZMPzRlAniSkthluc6GSVof-Jks5s--dKgaJpZM4Q8PII
.

Yeah, code splitting with SSR is pretty scary for all alternatives that exist. The only addition to core is exposing the prodStats to renderToHtml, and the Webpack stats has to be used by other popular alternatives like e.g. React Loadable.
So the fact that the example is complex isn't really an issue for react-static itself, I would argue. Exposing prodStats will not clash with a future potential code splitting solution in core either.

Can either of you see a nice way to achieve code splitting out of the box
for all users using this approach?
On Sun, Dec 10, 2017 at 8:03 AM Emil Tholin<EMAIL_ADDRESS>wrote:

Yeah, code splitting with SSR is pretty scary for all alternatives that
exist. The only addition to core is exposing the prodStats to renderToHtml,
and the Webpack stats has to be used by other popular alternatives like
e.g. React Loadable
https://github.com/thejameskyle/react-loadable#mapping-loaded-modules-to-bundles
.
So the fact that the example is complex isn't really an issue for
react-static itself, I would argue. Exposing prodStats will not clash
with a future potential code splitting solution in core either.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nozzle/react-static/pull/230#issuecomment-350554382,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFUmCRcJ8Q8DQSGkq_eWs-OQNh1Vthk5ks5s-_KngaJpZM4Q8PII
.

React Loadable, which is the least obtrusive code splitting library I know of, still forces you to render  your app in a certain way:
// src/index.js
import React from 'react';
import ReactDOM from 'react-dom';
import Loadable from 'react-loadable';
import App from './components/App';

window.main = () => {
  Loadable.preloadReady().then(() => {
    ReactDOM.hydrate(<App/>, document.getElementById('app'));
  });
};

And I'm not entirely sure it works with hot module reloading like react-static does, so I think @rileylnapier's example would be a great intermediary step until we find something obtrusive that can be added to core.

I personally think that it is good as it is right now: Expose stats to the user in renderToHtml, and let them do whatever they want with it, e.g. use react-universal-component like you've done in your great example. I don't think we should move all that to core and have react-static lock into that.

Fair.  I’ll work on fixing the readme and adding new example so we merge

Those changes you're working on should suffice.  Can't wait!


nixos/tiddlywiki: init

Motivation for this change
Service that runs TiddlyWiki nodejs server.
Exposed options to

enable basic auth
path to existing wiki files

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

The tests succeed and look reasonable. I don't see any other problems with this (though I don't have merge authority).

@GrahamcOfBorg test tiddlywiki

Please add the test to all-tests.nix.

just added, sorry missed that earlier

I've never used tiddlywiki before so pardon the my silly question... but how do I save? When I click the save button my browser attempts to download an html file.

Good question! Short answers is that wiki is saved automatically and save button is just for emergency.
TiddlyWiki has started as wiki in a single file that contains all code and wiki pages. Users supposed to open their local html file work with it and then save using save button.
It is very inconvinient so different ways to automatically syncronize wiki emerged https://tiddlywiki.com/#GettingStarted
TiddlyWiki node server is one of them. It deconstructs single html file into folder of tiddlers and serves them as webapp. This approach is more convinient for web hosting.
The save button in a context of node server becomes an emergency backup option. It allows to quickly backup everything into single html file again.

When I cycled the server my work was gone. I'm not sure the auto start is properly working.

Good catch! I've made a mistake in the wiki initialization command. I've added test to make sure wiki is persisted between restarts properly

User interface is a bit confusing. New files are open only until page is reloaded.
You can use search to find any wiki file. Or try Recent tab under search bar, it should contain recently created files.

So the trick about DynamicUser is that /var/lib/tiddlywiki actually maps to /var/lib/private/tiddlywiki. Your credentials file example may need some tweaking.

I understand your concern but I am not sure how to improve the example.
Credentials is relative path to dataDir https://tiddlywiki.com/#WebServer Parameter%3A credentials that is why in test on configured machine module can navigate from /var/lib/tiddlywiki to /var/lib/wikiusers.csv/ using ../wikiusers.csv credentials parameter.
The credentials parameter is not mandatory and admin can configure auth using either csv file or username/password https://tiddlywiki.com/#WebServer Parameter%3A username

@GrahamcOfBorg test tiddlywiki


Improve paper

I recently implemented the main algorithm in Factor with the paper as my main source.
My two main hurdles were these:

There is a typo Algorithm 5.6, steps 6–8: where there is 10^(k_0), it should be 10^(-k_0). I could only find this by looking at the reference implementation.
In Algorithm 5.4, step 5, or in the text, very little is said about how to break the tie. The code is a little obfuscated, the significand is 10s̃ + t, not f_c, as I first thought. Also helpful would be the information that you choose y^(rd) if the significand is odd, for the nearest-to-even case.

Another, smaller grievance is in section 5.2.3:

The information that d_1, d_2 are always zero is hidden away very well.
The upper bound for e in the integer check of x̃^(i) and z̃^(i) is a constant. It evaluates to 3 in the case of binary32 and binary64.

P.S.: In section 5.1.7 for completeness, you could mention that you compute r as r = z^(i) - 10*s. This might seem like baby steps, but I had to extract the obvious from another implementation.

Wonderful, many thanks!

There is a typo Algorithm 5.6, steps 6–8: where there is 10^(k_0), it should be 10^(-k_0). I could only find this by looking at the reference implementation.

Thanks, fixed.

In Algorithm 5.4, step 5, or in the text, very little is said about how to break the tie. The code is a little obfuscated, the significand is 10s̃ + t, not f_c, as I first thought. Also helpful would be the information that you choose y^(rd) if the significand is odd, for the nearest-to-even case.

I added a footnote about this in page 17 and made some comments on the reference implementation.

The information that d_1, d_2 are always zero is hidden away very well.
The upper bound for e in the integer check of x̃^(i) and z̃^(i) is a constant. It evaluates to 3 in the case of binary32 and binary64.

Maybe it's just me, but I tend to read all attached footnotes immediately after I complete reading a sentence, so I don't find it very much hidden. Also I think it's quite clear that $2^{p+2}-1$ and $2^{p+1}+1$ are constants, so it's also obvious that $d_{1},d_{2}$ are constants once the reader finds out their definition, so I think even if the reader missed the footnotes they could find out $d_{1},d_{2}$ easily by themselves. So I decided to not touch this part.

P.S.: In section 5.1.7 for completeness, you could mention that you compute r as r = z^(i) - 10*s. This might seem like baby steps, but I had to extract the obvious from another implementation.

Fair enough, edited accordingly. (I assumed you meant $r = z^{(i)} - 10^{\kappa+1}s$.)
Again, I appreciate it very much that you reviewed the paper. Few people did so far. I'm also quite glad that someone implemented this by actually reading the paper rather than copy-paste-modifying the reference implementation. Having multiple different implementations is a great way to cross-check both the paper and the implementations together. Please let me know if you find anything unclear or suspicious or erroneous afterwards, thanks!

In Algorithm 5.4, step 5, or in the text, very little is said about how to break the tie. The code is a little obfuscated, the significand is 10s̃ + t, not f_c, as I first thought. Also helpful would be the information that you choose y^(rd) if the significand is odd, for the nearest-to-even case.

I added a footnote about this in page 17 and made some comments on the reference implementation.

I think that helps a lot.

The information that d_1, d_2 are always zero is hidden away very well.
The upper bound for e in the integer check of x̃^(i) and z̃^(i) is a constant. It evaluates to 3 in the case of binary32 and binary64.

Maybe it's just me, but I tend to read all attached footnotes immediately after I complete reading a sentence, so I don't find it very much hidden. Also I think it's quite clear that 2p+2−1 and 2p+1+1 are constants, so it's also obvious that d1,d2 are constants once the reader finds out their definition, so I think even if the reader missed the footnotes they could find out d1,d2 easily by themselves. So I decided to not touch this part.

It's not directly hidden, but I think the reader could benefit if the footnotes were included directly in the text, at the end of the section, with the addition that for binary32 and binary64, the upper bound evaluates to 3 or that x is an integer iff 2 ≤ e ≤ 3 and z is an integer iff 0 ≤ e ≤ 3.

P.S.: In section 5.1.7 for completeness, you could mention that you compute r as r = z^(i) - 10*s. This might seem like baby steps, but I had to extract the obvious from another implementation.

Fair enough, edited accordingly. (I assumed you meant r=z(i)−10κ+1s.)

You are of course right.


Can't set hostname in The Forest

when starting a forest server, the name is always 123.
Thanks

The hostname is set via your startup .bat script. It has never been "123" as a default, and is actually "My Oxide Server" in the example .bat we include.

Thanks this is what i have set.
TheForest.exe -batchmode -nographics -ip <IP_ADDRESS> -port 27016 -maxplayers 10 -hostname "test server" -friendsonly 1
though this also seems to create multiply servers. e.g. the server list populates with about 10 or so servers callerd 123.
Thanks for your help so far, really quick responce

p.s. i was using the latest snapshot if that makes a difference

I fixed the issue, i found out that if i created a server through the forest using host then quit the server, the oxide server details were overriden by  the details that, i had previously entered.
so i have entered all the details using the host feature in the game and now  it works perfectly when i quit and re-host using oxide.
Thanks again


avahi-daemon

Hi Thomas!
Great work with openHABian, I've tried it with a spare Raspberry Pi 3 that was lying around and everything went perfect! Awesome!
But one thing that I missed (and that it is preinstalled on Raspbian) is to be able to get the Raspberry Pi's IP address without connecting a monitor to it, just using avahi-discover. For that, you only need to install the package "avahi-daemon" to the distribution and then any device on the network can get the its IP address through avahi-discover. Even more, if the Raspberry Pi has the avahi-daemon installed, you can establish an SSH connection just typing "ssh<EMAIL_ADDRESS>I know that the package can be installed after the first ssh connection with "sudo apt install avahi-daemon", but IMHO it's worth adding it to the distribution.
Many thanks again for all your hard work and best regards,
Aitor

Hey @bodiroga, I know the name and have no problem in adding it. However, can you point me to an article describing the benefits in more detail? I'm already successfully resolving:
$ ssh openHABianPi
pi@openhabianpi's password:
...

Hi @ThomDietrich !
Many thanks for your fast answer!
Unfortunately, I have reinstalled openHABianPi on my Raspberry Pi to try your suggestion, but by default my laptop doesn't resolve the hostname:
aitor@aitor-portatil:~$ ssh openHABianPi
ssh: Could not resolve hostname openhabianpi: Name or service not known
aitor@aitor-portatil:~$ ping openHABianPi
ping: unknown host openHABianPi
Let's see if anyone can jump in and tell us if they can access their Raspberry Pis just by the hostname.
Best regards,
Aitor

Probably this is the reason why my laptop can't resolve the Raspberry Pi's hostname to an IP address:
https://www.raspberrypi.org/forums/viewtopic.php?f=51&t=46264
As you can see in the third comment, avahi-discover allows network devices to be discovered even if a DNS server is not present in your local network.
Up to you Thomas, you decide if you want to include it in the distribution ;) The only thing I can tell you is that it's shipped by default in both Raspbian and Raspian Lite ;)

As you can see in the third comment, avahi-discover allows network devices to be discovered even if a DNS server is not present in your local network.

That part is clear to me, I just though every router does that nowadays :) Okay I guess it's decided. I'll include avahi-daemon. Anything else you whould like to see preinstalled? (list of installed software)

Hi Thomas!
It's great to see that you have decided to include the program in the distribution, I'm sure that some new users will welcome your addition ;)
Many thanks for all the work you are doing on openHABian, it's never been so easy to have a working openHAB enviroment!


Mismatch between AlphaPose generated keypoints format and retiming code expected format

Hi. I'm interested in running inference on my own video.
I've followed the instructions and got stuck at step 3 which specifies: "Run AlphaPose and Pose Tracking".
I've ran AlphaPose with tracking (knowing the exact tracking configuration command you have used would be very useful by the way).
None of the data formats which have been specified in the official AlphaPose output formats is the one your code expects.
Your code expects keypoints in format [[x1,y1,1], [x2,y2,1] ....], but none of the output formats in the document I've shared supports that.
Could you please share how to do the conversion? Or perhaps a more detailed process of how to generate keypoints in this format?

Hi, sorry for the confusion. The tracking used is this one: https://github.com/MVIG-SJTU/AlphaPose/tree/master/trackers/PoseFlow#for-any-datasets-general-version
In that example, the alphapose-results-forvis-tracked.json is the expected input to our code.

Thanks. That solved my issue :)

Hi, I got same problem with AlphaPose.
Did you run the scripts/demo_inference.py and then run the trackers/AlphaPose/tracker-general.py to get the right JSON format??