Vectorized Gen.normal distribution

Summary
This PR implements Gen.broadcasted_normal, which is like Gen.normal except that the parameters mu and std are broadcast-compatible arrays, and the samples from this distribution have shape Broadcast.broadcast_shapes(size(mu), size(std)).  The distribution represents a random Array of independent normals.  This is equivalent to (a reshape of) a multivariate normal with diagonal covariance matrix, but its implementation is more efficient than that of the more general Gen.mvnormal.
One common use case this distribution serves is modelling IID Gaussian noise.

Comment from @alex-lew:

Nice! This makes sense to me as long as you never need to constrain individual elements and leave others unconstrained — for that case, the Map combinator is the solution (it transforms a generative function to take vector versions of its arguments, and maps the original function over the vectors). In general, before merging this, it seems good to think about what Gen’s relationship with tensors / broadcasting will be — seems a little strange to support this on Normal but not other distributions? But maybe it’s fine!

Just re-read Alex's above comment

This would introduce two different ways of making groups of independent choices: one in which they are all considered “one choice” (a single trace address), and one in which they are given separate addresses (the Map combinator). It seems the first is strictly less flexible than the second? It seems that long-term, the right thing might be to add better (more efficient) support for the second one, so that it is more usable for this type of use case, than to add the first one as an option.

and I totally agree with that.

Now, I'm much more sympathetic to cases like fill(1) .+ fill(2) or fill(1) .== 1 because there you are already starting with an Array{T,0}, and there is some good sense in preserving that. But in the case of 1 .+ 2 I'm more strongly opposed to wrapping its result in an Array{T,0}. You are starting with something that has axes and supports getindex and you end up with the same sort of thing that has axes and supports getindex. The result is also far more useful.

I would describe the situation as this: There is a type U and a function
f(w::W, x::U, y::U)::U

(in our case f = broadcast).  There is an implicit conversion from Float64 to U.  We then write z = f(w, 1.0, 2.0).  I think we don't have any reason to expect z to have all the same behaviors as Float64 unless there is also an implicit conversion from U to Float64.  (Also, Array{Float64,0} does support axes and getindex?)

Constraining individual entries of the matrix is (AFAICS) fundamentally incompatible with broadcasting.

@bzinberg Can you explain your thinking here a bit more clearly? Broadcasting is about the shapes of the parameters being broadcast to determine how many random samples to make. I don't see how that is incompatible with posing an inference query like, "suppose the fifth random sample was 2.3"?

There is a very common use case for the version implemented here: IID noise (where we don't propose directly to the noise value, but instead propose to a RV that serves as the mean of that noise).

You don't propose to the noisy value, but you constrain it (with observations) -- yes? If you only observed part of the matrix, you'd presumably want to constrain only part of the matrix. Of course, there are cases (presumably including Cora) where you always observe all of the matrix.

It's not clear to me that we should expect the overhead of large numbers of traced quantities to be negligible. If we do, that seems like a separate project that should not block us at least having a broadcast option available in the medium term.

Hmm, we can definitely discuss further, but I'm relatively confident that we can achieve little or no overhead. Note that in your version, all the quantities are also "traced" -- just not at separate addresses. I imagine we could keep the exact same memory footprint, but add an addressing scheme for addressing elements of iid random arrays. For example, we could implement a IIDBroadcastedDistribution{T} generative function type, whose trace (like the Map combinator's trace type) is just an array, but whose trace and choicemap types have special implementations of accessor methods (like get_choices and getindex) that interpret addresses as indices into that array.

Looking at this a second timeli, I think this is simpler than I originally realized.
The only thing that's incompatible with broadcasting is broadcasting x together with mu and std -- if we require x to have shape exactly equal to broadcast_shapes(size(mu), size(std)), then this just just another distribution following the same API as the existing ones -- it just happens to take its parameters in a compact form and do an efficient computation.  And luckily, that is (AFAICS) not something anyone would ever want to do -- I don't see any usefulness of broadcasting the observed value of a normal.  This may have been obvious to you and that's why you were confused :slightly_smiling_face:
Agreed that this distribution should have a separate name like BroadcastedNormal (or maybe BroadcastedIndependentNormal), at least until we decide how to handle vectorization in Gen (which is longer term and does not need to be decided now).
Very cool about achieving negligible overhead and a more granular addressing scheme!  It's not strictly needed for this PR since we're requiring x to have full shape, but I'll definitely be interested to see this develop.
I'll make the changes above and then I think this is pretty safe to merge, WDYT?

Done implementing https://github.com/probcomp/Gen/pull/102#issuecomment-495963392, I think this is ready to merge.

Hi @bzinberg -- hmm, I wonder if we are talking past each other a bit?
My comment isn't about broadcasting, but about vector-valued distributions.
There is a question of whether to have vector-valued distributions, or generative functions with efficient trace implementations that return vectors and expose a list of random choices at indexed addresses.
I'll leave you and Marco to discuss merging this PR (I don't think there's any harm), but I'm a little hazy myself on when this (or any other custom distribution) goes in "Core Gen" vs. whatever project is using it (it's totally kosher for a Gen project to define a custom distribution), and suspect that long-term, the generative-function approach might make more sense than the vector-valued-distribution approach.

@alex-lew Ah, I see.  I was completely not aware of that concern -- I thought the existing Gen.mvnormal distribution was already vector-valued, hence I thought some version of that API decision was already made.

Yeah, mvnormal is the one current example of a vector-valued distribution. Before moving further in that direction, I wanted to express these concerns, as I think that in many cases, it is more natural to think of multivariate distributions as generative functions with efficient implementations. I would be in favor of making mvnormal a generative function, if we could get the overhead down, so that we could build in the ability to condition on some elements of the vector (especially since, as in the IID case, conditioning a multivariate normal is tractable to do exactly).

Ah, I didn't realize that Gen.mvnormal was the only existing example and the design of vector-valued distributions-or-should-they-be-GFs is still up in the air.  I totally get why you wanted to hold off until further discussion is had - and would be great to have you, me and @marcoct meet and discuss this when we have time.  Thanks for sharing your concern!

It looks like broadcasted_normal works for all scalar arguments, subsuming the functionality of normal. To reduce the duplicated logic of normal and broadcasted_normal, I was going to suggest replacing normal with the broadcasted_normal implementation, but: (1) the behavior of broadcasted_normal on all scalars depends on the behavior of .* on two zero-dimensional arrays, which seems a little brittle,

Yeah, I agree.  Alex and I discussed this above in https://github.com/probcomp/Gen/pull/102#issuecomment-495480617.  The zero-dimensional case honestly is a pretty significant Julia issue in my opinion, and I participated in the discussion at some length in https://github.com/JuliaLang/julia/issues/28866.  I slightly prefer keeping the separate name for broadcasted_normal, because of (i) Julia's inconsistent conflation of scalars and zero-dimensional arrays, and (ii) makes it less of a breaking change if/when in the future we decide to change the design of multivariate distributions.

and (2) there is a performance cost:
@time for i=1:10000 broadcasted_normal(0, 1) end
  0.000376 seconds (10.00 k allocations: 937.500 KiB)

@time for i=1:10000 normal(0, 1) end
  0.000067 seconds

Note that the performance difference comes from randn() versus randn(()).

Nice catch!  Didn't realize.  (I wonder why there's such a difference...)

Regarding putting this in Gen vs out of Gen, this vectorized normal is used often enough within inverse-graphics style projects that it seems useful to have it built-in. Let's keep this as part of Gen for now. However, I think it would be natural to include broadcasted versions of the other primitive distributions, and to do this in a way that avoids duplicated logic and lots of boilerplate code.

Agreed with all the above.

For example, it seems possible to at least auto-generate the test code for each new vectorized distribution.

Cool idea!

My one request is to add a vectorized Bernoulli as part of this pull request, in order to help identify how the implementation pattern generalizes to other distributions, and to identify low-hanging fruit opportunities for automation and reduction of boilerplate.

[Response requested] I would prefer to do this in a separate PR, because the current content of this PR blocks my Cora work (unless I patch it in, but seems like that's not necessary at this point).  I will do the PR but Cora is more time sensitive at the moment.  Sound ok?

Regarding longer term plans for multivariate distributions in Gen -- it is true that generative functions for certain common multivariate distributions in which random variables are dependent on other variables would be useful. Examples include multivariate normal, and exchangeable distributions like normal-inverse-gamma multivariate distribution (collapsed mean and variance of the normal). However, these are subtly different from one another in their properties (e.g. exchangeability is a special property that the multivariate normal does not possess), and they are both different from the special case handled here in which all variables are independent. I suspect that achieving the same performance using a more flexible generative function will not be possible or at least would require much more complicated code. I think that simple vectorization of primitive distributions as done here is a sufficiently different use case to warrant a distinct implementation strategy.

Definitely interested in discussing this more.

BTW, regarding the performance differences, it looks like the the difference comes not from the sampling itself but from the packaging of scalars into 0D vectors:
julia> @time for i=1:100_000 randn(()) end
  0.004398 seconds (100.00 k allocations: 9.155 MiB)

julia> @time for i=1:100_000 randn() end
  0.000458 seconds

julia> @time for i=1:100_000 fill(randn()) end
  0.005900 seconds (100.00 k allocations: 9.155 MiB, 38.47% gc time)

Sounds good :)

Regarding longer term plans for multivariate distributions in Gen -- it is true that generative functions for certain common multivariate distributions in which random variables are dependent on other variables would be useful. Examples include multivariate normal, and exchangeable distributions like normal-inverse-gamma multivariate distribution (collapsed mean and variance of the normal). However, these are subtly different from one another in their properties (e.g. exchangeability is a special property that the multivariate normal does not possess), and they are both different from the special case handled here in which all variables are independent. I suspect that achieving the same performance using a more flexible generative function will not be possible or at least would require much more complicated code.

Interesting. Not an urgent discussion to have now, but I'd have thought that an IndependentChoices generative function (parameterized by a scalar-valued distribution) could use essentially the same logic (and therefore achieve the same performance) as a vector-valued distribution. But it's certainly possible I'm missing something.
I thought such a feature might be useful even in the (conditionally) independent case. Broadly, the two use cases I imagine are:

Although the vector of variables are themselves independent, a downstream consumer of the vector makes them conditionally dependent, given an observation of some downstream variable (e.g., normal(sum(xs), 1)).  In this case, a proposal may wish to propose individually to elements of the vector, rather than have to propose to every element at once. If the vector is a distribution, this can only be done with involution MH.

Suppose we draw a vector of (latent) parameters from a multivariate normal, then use them to parameterize a vector of conditionally independent draws from some 1-parameter distribution. Observing only some of the variables allows us (via MH or importance sampling, e.g.) to make inferences about the others.

If it really isn't not possible to do this with no performance overhead, I agree that it fits Gen's philosophy to provide both ways of doing it--a more performant way for when you are happy to think of an entire vector as a single choice, and a slightly less performant way for when you need the elements to be separately addressed. (And again, we needn't finish this conversation now.)

@alex-lew Your suggestion of IndependentChoices generative function constructor, which takes in a distribution, is very interesting.
Regarding performance, the GFI methods for the resulting generative function would need to be able to accept a more general class of arguments than the vectorized distribution version. For example, its regenerate method would need to handle arbitrary subsets of indices inside selections, and its generate would need to handle arbitrary constraints. To key challenge is to see whether we can dispatch to special code that is optimized for the special case handled by the vectorized distribution handles where all addresses are selected / constrained. For selections, this is possible using the AllSelection. For constraints, this could be plausibly implemented by introducing a special type of choice map like a DenseVectorChoiceMap that indicates, in its type, that it represents a dense array of addresses. That's possible, but would involve introducing another type that users would need to know about. One way around introducing another user-facing type would be to define a special behavior for choice maps that contain Julia Arrays as their values. Then, a user could populate the constraints choice map with :v => Float64[1.5, 2.3, 3.143] and this would get automatically get converted to a DenseVectorChoiceMap(Float64[1.5, 2.3, 3.143]) before being passed to the generate method of theIndependentChoices generative function. One would then also need to ensure that when simulating or proposing complete traces from an IndependentChoices generative function, the fact that that it a full dense set of addresses needs to be maintained. This should also be possible. But both of these require some careful thinking about the more general problem of how to annotate choice maps with type information that conveys extra information about their structure, which is also something that the Static modeling language does. These two cases should probably be considered together when thinking about this more general problem.
Another potential performance issue is that in order to efficiently implement the GFI, the trace of an IndependentChoices would need to be a persistent data structure (e.g. backed by a PersistentVector and not a simple Array).

@marcoct Thanks for writing that out -- that all makes sense. Yes, I had imagined dispatching on whether the argument was a full vector or not.
Regarding:

Another potential performance issue is that in order to efficiently implement the GFI, the trace of an IndependentChoices would need to be a persistent data structure (e.g. backed by a PersistentVector and not a simple Array).

Currently, are the arrays stored in traces (of mvnormal, e.g.) immutable? If I write an involution proposal that extracts an mvnormal and changes pieces of it, then propose the changed vector, might I have accidentally changed the original trace?

Currently, are the arrays stored in traces (of mvnormal, e.g.) immutable? If I write an involution proposal that extracts an mvnormal and changes pieces of it, then propose the changed vector, might I have accidentally changed the original trace?

They are currently not immutable. They are currently Vector{Float64}s.  Yes, it would probably be better if they were wrapped in an immutable type that does not provide setter methods.

Marco and I spoke about this, and agreed that I'll patch this distribution into Cora for now, and will attempt over the next 2 weeks to do the request in https://github.com/probcomp/Gen/pull/102#pullrequestreview-242751246 (depends how much time I am able to spend on it).  If it takes longer than 2 weeks then would be better to merge as-is.

Discussions on broadcasting from two other Julia PPLs, if we revisit this in the future:
https://github.com/cscherrer/Soss.jl/issues/14
https://github.com/TuringLang/Turing.jl/issues/476


⚠ Bofrost.de has degraded performance

In 0674410, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1660 ms

Resolved: Bofrost.de performance has improved in 0b2a855.


ceph-iscsi: don't use ceph_dev_xxx variables

Using ceph_dev_branch and ceph_dev_sha1 for configuring ceph-iscsi
repositories from shaman doesn't make sense because the ceph devel
branches and sha1 aren't compatible with ceph-iscsi devel.
Instead we could rely on the master branch and the latest sha1.
Currently it's not possible to using a custom ceph branch/sha1 value
with iscsi setup otherwise the repository setup will fail.
Signed-off-by: Dimitri Savineau<EMAIL_ADDRESS>

jenkins test centos-non_container-lvm_osds

jenkins test centos-non_container-lvm_batch

jenkins test centos-non_container-all_daemons


make a website for the project

It would be nice to have a website for this project with documentation, getting started guide, tutorials, etc. It could be kept in this repo and hosted with GitHub pages.

Looking into Material for MkDocs for a documentation site - https://squidfunk.github.io/mkdocs-material/


Add integration tests for soci index rm command

Signed-off-by: Jin Dong<EMAIL_ADDRESS>Issue #, if available:
Part of #54
Description of changes:
Add 3 integration tests for soci index rm.
Testing performed:
make test && make check && make integration
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Should the integ tests be designed to fail with the bug #251 still open?

Should the integ tests be designed to fail with the bug #251 still open?

Ideally yes.
The integration tests in this PR are only based on the current behavior. E.g., 1/ soci index rm only deletes metadata (so not shown in soci index list) but not delete in storage (so soci index info still work); 2/ soci index rm invalidString doesn't retrun error or print error message.
When we fix #251, we should add extra checks that check the removed index is also removed from storage (e.g., soci index info indexDigest returns error, or check the blob storage).
Similarly, when we fix #252, we should add an integration test that check soci index rm randomStringOrDigest should return an error.
These extra checks / new integration tests will be addition to the integration tests in this PR.


Chrome: side bar profile image doesn't load due HTTP 429 response code (Too Many Requests)

Hi,
TweetDeck-Enhancer side bar does not load the profile image and data due an error fetching data,  always asks for waiting or refreshing the page:

Debugging it deeper, it seems to be an error caused because the extension is always getting HTTP 429 response code (Too Many Requests):

Too many requests is a Twitter API issue beyond my control, so I can't do much at all to fix that.


Add Fourier Feature Mapping for Improved Quality

A pal of mine discovered a fantastic notebook by a coder using the Github handle eps696. The notebook adds fourier feature mapping to CLIP+SIREN to great effect.
Here's the Original Notebook and here's
My Custom Version with some fun additions such as prompts to minimize (subtract) and a prompt for painting finer details. All due credit, those ideas are also from eps696 in another of their notebooks.
Anyway, it definitely seems to help. Using as few as 16 SIREN layers, I've gotten this output:
cosmic love and attention
https://user-images.githubusercontent.com/3994972/109033636-77a91200-768c-11eb-8d5c-265f745cf496.mp4
mist over green hills

mist over green hills with the fine_details="trees"

The drawbacks seem to be that the learning rate is much more fickle and may even need to be changed slightly on a per-phrase basis (if you get really unlucky). Increasing the number of SIREN layers to anything more than 20 causes lots of issues as well and I'm pretty unclear as to why that is. I can't find a stable learning rate for those. Also, there is a fourier_scale parameter which eps696 left at 4, but I've found 2 to be a better result.

Can you explain what it does, precisely? Looks nice! It would be interesting to compare it to the generations of the new size schedule of #61 and to combine it.

I can not! Unfortunately my expertise in machine learning is quite limited. I just found the notebook and saw the generations were indeed quite nice. Looking at your output though, I think the primary contributor to the quality of the latter image is just that I have the fine_details prompt enabled. It's probably worth looking into implementing that feature as well.
I'll do some comparisons when I get time, but for now I think you've "solved" this issue through other means.


Implement evolution check

Should just loop through party Pokemon and see if any should evolve

With a little more granularity: I think it'd make sense to have one method that takes in a Pokemon and returns whether it should evolve (and to what).

With a little more granularity: I think it'd make sense to have one method that takes in a Pokemon and returns whether it should evolve (and to what). You'd later be able to loop over the Pokemon in a party and run operations depending on the result of this method.


Return error on any issues with curl

By default (HTTP) Fail silently (no output at all) on server errors.
This is mostly done to better enable scripts etc to better
deal with failed attempts.
So we must fail if some resources are unavailable

LGTM


Handle slate to MDX for custom blocks

Related to #14

Going to close this since its mostly addressed in #16


Adding max_analyzed_offset option for highlighting

NEST/Elasticsearch.Net version: 7.7.14
Elasticsearch version: 7.6.1
.NET runtime version: 6.0
Operating system version: Azure Web App
Description of the problem including expected versus actual behavior:
Same issue logged here https://github.com/elastic/elasticsearch-py/issues/1719
I have the following code:
var request = new SearchRequest<ElasticDocument>()
{
    Query = new BoolQuery()
    {
        Must = must,
        Should = should,
        MinimumShouldMatch = 1
    },
    Sort = sorts,
    From = from,
    Size = size,
    Highlight = new Highlight()
    {
        NumberOfFragments = 1,
        Order = HighlighterOrder.Score,
        Fields = new Dictionary<Field, IHighlightField>()
        {
            [Infer.Field<ElasticDocument>(f => f.Body)] = new HighlightField()
        },
        PreTags = new[] { "<b>" },
        PostTags = new[] { "</b>" },
        NoMatchSize = 100,
        MaxAnalyzedOffset = 1000000
    },
};

Which triggers the error:
"Request failed to execute. Call: Status code 400 from: POST /document/_search?typed_keys=true. ServerError: Type: x_content_parse_exception Reason: \"[1:45] [highlight] unknown field [max_analyzed_offset]\""

Expected behavior
Highlighting works with no error for long documents

Hi @catmanjan. The max_analyzed_offset options was not added until Elasticsearch 7.12 and is not supported on the version you are using. This error from the server is therefore accurate and expected. To support using this feature, you would need to upgrade your cluster. I'll close this issue on that basis as this appears not to be a fault with the client.


Added limit(1) to exist query

What exactly is this trying to fix?  This doesn't really change any behavior.  If you look at the getKey() implementation, you'll see that a limit of 1 is already being transparently applied.


Web/Android support

Access to iCloud is possible via web browser.  It would be a great enhancement if you could also use iCloud on the web and on Android via web access.

Hello.
I really want that feature. However, it looks like Apple does not provide an API to access iCloud Documents from non-Apple devices.

https://developer.apple.com/library/archive/documentation/General/Conceptual/iCloudDesignGuide/Chapters/Introduction.html#//apple_ref/doc/uid/TP40012094-CH1-SW1
According to the official Apple documentation, Apple provides three means for developers to integrate with iCloud.

Key-Value Store
iCloud Document Storage
CloudKit

The Key-Value Store and CloudKit services cannot directly access the user's iCloud Document folder. Instead, these services allow storing app-specific data in the user's designated iCloud storage area.
The iCloud Document Storage option enables accessing the user's iCloud Document folder, but this access is limited to Apple's devices running iOS, iPadOS, or macOS. The integration happens through the respective operating system's APIs, not a cloud-based API.
Therefore, while Apple provides ways to leverage iCloud for data synchronization and storage within apps, there is no publicly available cloud API that allows direct programmatic access to a user's iCloud Document folder from outside of Apple's device ecosystem.


[Backport] Add avro dependency to parquet extension

Backport of apache/druid#9124 to 0.17.0-iap

We'll need to backport https://github.com/apache/druid/pull/9120 to 0.17.0-iap, the tests are failing on the  issue fixed by that

I opened the backport at https://github.com/implydata/druid/pull/29


thumbnail in blog not showing up

Hi. thanks so much for this theme! it's great!
I have intermittent success with thumbnail showing up in blog section.
/content/blog/_index.md
show_post_thumbnail: true
thumbnails and named with "feature", and placed on the directory of the blog
url: https://www.kenkoonwong.com/blog/
github: https://github.com/kenkoonwong/everyday_is_a_school_day/tree/main/content/blog
it works fine with blogdown but for some reason when using netlify, it does not seem to like it. Thanks!

nevermind! it was a netlify deployment issue. I logged onto netlify and retry deployment and it worked.

@kenkoonwong I am having the same issue, and it looks like from your first URL that the second image in the list doesn't render. If you fixed it somewhere else, did you change anything during redeployment to fix your issue? On redeployment, it seems like a random image is chosen to not render.

The behaviour is random, see also #94. I posted on the netlify forums to see if they could provide some support: https://answers.netlify.com/t/netlify-creating-random-changes-to-hugo-site-on-deploy-missing-content/79960
@kenkoonwong could you please reopen this issue for visibility?
@bryceikeda what pages are you having this issue on? I noticed it on my project page, but it sounds like you and @kenkoonwong have it with your blog listings?

@kenkoonwong I am having the same issue, and it looks like from your first URL that the second image in the list doesn't render. If you fixed it somewhere else, did you change anything during redeployment to fix your issue? On redeployment, it seems like a random image is chosen to not render.

I haven’t found a solid fix. Anytime when this happens I’ll hv to redeploy and hope it fixes it

@kenkoonwong I am having the same issue, and it looks like from your first URL that the second image in the list doesn't render. If you fixed it somewhere else, did you change anything during redeployment to fix your issue? On redeployment, it seems like a random image is chosen to not render.

I had to clear cache and redeploy twice to get all the pictures to load

The behaviour is random, see also #94. I posted on the netlify forums to see if they could provide some support: https://answers.netlify.com/t/netlify-creating-random-changes-to-hugo-site-on-deploy-missing-content/79960
@kenkoonwong could you please reopen this issue for visibility?
@bryceikeda what pages are you having this issue on? I noticed it on my project page, but it sounds like you and @kenkoonwong have it with your blog listings?
Edit: Also, the behaviour is still affecting your site @kenkoonwong. On the current deploy this post is missing the thumbnail and author byline even though both should be displayed:

I had to clear cache and redeploy twice to get all the pictures to load

@mccarthy-m-g Yeah mine is on the blog page.
@kenkoonwong I've tried doing that but I always have at least one picture that does not show up...

@bryceikeda What layout are you using for your blog page? (the layout param in _index.md) And what Hugo version?
I’ve found redeploying over and over eventually lands on a build where nothing is missing, but doing that isn’t sustainable.
A Netlify support engineer responded to me on the Netlify forum post I shared earlier, so hopefully they can help.

@mccarthy-m-g I am currently using version 0.89.4 and the list-grid layout.

Thanks! Just some notes for reference:
Hugo versions
I'm using version 0.89.2 and Ken is using 0.102.1. So we're having the same issue across three different Hugo versions.
summary-grid
For Bryce and I the following If statements are randomly being evaluated as false on our blog and project pages respectively during the Netlify build, causing the content associated with them to not show up. This behaviour only happens if the respective show_ params are set to true in _index.md.
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/partials/shared/summary-grid.html#L7
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/partials/shared/summary-grid.html#L31-L33
The summary-grid partial is called here for blog pages with the list-grid layout:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/blog/list-grid.html#L13
And here for project pages list-grid layout:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/project/list-grid.html#L13
summary-list
For Ken the following If statements are randomly being evaluated as false on Netlify builds:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/partials/shared/summary.html#L7
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/partials/shared/summary.html#L35-L37
The summary partial is called here for blog pages with the list-sidebar layout:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/blog/list-sidebar.html#L8
summary-compact
One thing I've found on my site is that this behaviour does not happen on my talks page. For talks the summary-compact partial is used for listings, which is called here with the list layout:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/talk/list.html#L17
And here with the list-sidebar layout:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/talk/list-sidebar.html#L13
Partials and layouts
As far as I can tell, the partials are all using the same Hugo code for If statements. So if the bug is caused by the theme and not Netlify, I don't think problem is there.
However, the layouts that call these partial range over the pages slightly differently. Blogs/projects:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/project/list-grid.html#L9-L15
Talks:
https://github.com/hugo-apero/hugo-apero/blob/17f92f68899f672868c8f1e93a5b7d26232cf11d/layouts/talk/list.html#L8-L22
My Hugo is rusty so I'd have to go back to the docs to understand this better, but there could be a weird interaction happening on Netlify when ranging over the pages that's causing the random missing content.

@mccarthy-m-g So would Edit 2 be the current workaround until the true bug is fixed? If so, (I am not well versed in Hugo) is it as simple as setting {{ if $root.Params.show_post_thumbnail }} in summary.html to {{ if true}}?

@bryceikeda Yeah, that’s the easiest workaround at the moment. I would do it for all the if statements though. Don’t directly edit the file inside the theme directory, instead override the partial by creating a layouts/partials/shared/summary-grid.html file where the path starts from your root directory.

@mccarthy-m-g Worked! Appreciate your help on this!

@bryceikeda @mccarthy-m-g pardon my noobness in Hugo. What do I need to do to workaround the version I’m using? Thanks!

See this page for more info on how overriding theme files works: https://bwaycer.github.io/hugo_tutorial.hugo/themes/customizing/
You just need to override the appropriate theme file for your site, then replace the contents of the If statements with true. See the posts above for the file and If statements in question.

summary-grid

Hi @kenkoonwong to be more explicit (I too sometimes need clearer directions), here are the steps on what I did.

Go to themes/hugo-apero/layouts/blog (this is the page layout I was using that was not showing pictures correctly)
Click list-grid.html (this is the layout that I am using)
Here you can see it is using {{ partial "shared/summary-grid" . }}
As stated by @mccarthy-m-g, don't directly edit the file inside the theme directory, instead override the partial by creating a layouts/partials/shared/summary-grid.html file where the path starts from your root directory. You can find this summary-grid.html file in the themes/hugo-apero/layouts/partials/shared/ directory.
In this new summary-grid.html file, right below where it says <!--if show_post_thumbnail is TRUE in root--> update it to say {{ if true }}

Then your website should recompile and render the pictures!

@bryceikeda @mccarthy-m-g Thank you so much !!! Made the change as instructed and so far working very well!!


can you supply the inference code?

1，Whether the reinforcement learning algorithm mentioned in the article needs to use the data generated by Safebench for training？
2，Can you provide inference code and pre training model ？such as DQN and DDPG

Hi,
Thank you for your interest in our work!

The RL models used in our paper are trained with data in SafeBench. However, you can test your models trained with any other data. They don't necessarily need to be trained with SafeBench.
You can download the models used in our paper here. We also updated the README. Please follow the instructions in step 4 to evaluate the models.

Please let me know if you have further questions!


Investigate performance issues reported for TS Server

Various issues tracking performance concerns:
#21442
#21292
#10439
#17565
#20260
#19997
#18774

The major finding from the investigation is this bug: https://github.com/Microsoft/TypeScript/issues/14636
This effects any workspace that many folders. Adding an exclude or include to the jsconfig.json or tsconfig.json fixes this in my testing, but we really need a fix on the TS side. No logs or errors are generated in these cases
For other cases, #22915 should make investigation easier as well
Closing out this investigation work item. Will continue following up on the individual performance issues


log4rs hides all information about its appenders

I've discovered that it's currently impossible to retrieve any information about appenders. My case is a web server that should be able to give all logs on request. So I need look through log4rs::config::Config, find log4rs::append::file::FileAppender and get path to file it holds.
My suggestion: to cover this case, log4rs::append::Append should support downcast mechanism and all appenders should provide getters for their data.

@survived thanks for opening an issue. To clarify, you're not looking of all logs that the appender has written correct? You're just looking for the config file right? I would be happy to help you push a pull request through. Maybe you can start by having a look through the appender code and propose a change to the public api?

@estk Thanks for responding! Yes, you're right, I'm looking for the way to find out if FileAppender is present in log4rs configuration and get the path of file. I'll make all changes in api (I believe they're trivial) and ask you for review.

Closing in favor of the more general #102


Add Tooltip Translations for the buttons.

I added tooltip translations for languages that are already included, if u approved my PR for dutch.
Then just add this under the translations for nl.
ToolTipPreviousYear: "Vorig jaar",
ToolTipNextYear: "Volgend jaar",
ToolTipCloseSidebar: "Sluit de zijbalk",
ToolTipCloseEventList: "Sluit de event lijst"
I also roughly translated it from google translate.
So i'm not sure if everything is 100% correctly translated
For my project every little thing has to be translated for screen readers.
Regards

Hello @joristackaert, I just resolved the merge conflict and change it a bit, but thank you for this thing!


Only allow for cross origin requests on localhost

Acceptance Criteria

I want to protect certain routes from postman/curl requests to the server

Notes

At some point, I want to create an actual api for this blog for the public to consume

Fixed


Support pkg created in pg_catalog to call pkg functions without pg_ca…

#88

With GetOverrideSearchPath/PushOverrideSearchPath we don't need to add the pg_catalog to schema list explicitly, because it get's added in PushOverrideSearchPath.
perhaps we should move get_package_oid() to namespace.c file, so we don't have to fetch search path again after GetOverrideSearchPath/PushOverrideSearchPath. And we can loop over activeSearchPath directly.

Hi @rasifr, thank you and your comments. I will update my pr with this approach.


Feature request: numpy arrays of objects as inputs to dask.compute

Currently, I can create (nested) lists of objects that are a mix of eagerly computed items and delayed items.
If I pass that list to dask.compute, it can create the graph and computes the result as a new list replacing the delayed items with their computed counterparts.
The list has a very well defined structure that I would like to exploit. As such, before using Dask, I had been using numpy array with dtype=object.
The workaround, suggested by @mrocklin, is something like:
tasks = np.asarray(dask.compute(*tasks.tolist()))

Are there any plans to support numpy arrays as inputs to the compute function?
I understand it probably isn't an easy answer due to the fact that you should support nesting and whatnot.
Code example
Stack overflow question answered by @mrocklin Thanks!

Not sure if this makes sense for your use case, but there are a number of ways to join Dask Arrays together into a single Dask Array. It's also possible to convert Delayed objects to Dask Arrays. NumPy Arrays can also be converted, but that conversion often happens under the hood. Also asarray can be used to coerce various array-like things including Python lists to Dask Arrays. Would be interested to hear if this helps or not.

@mrocklin, see PR that is referenced above. I check if the numpy array is a numpy array of objects first. In which case, it is basically equivalent to a dictionary (which may be arbitrary large as well).

Right, I think that the same logic holds.  It's much more likely that that
numpy array of objects holds a bunch of text.  Does this approach
significantly slow down that case?  My guess is that it does.
On Sat, Jun 30, 2018 at 5:28 PM, Mark Harfouche<EMAIL_ADDRESS>wrote:

@mrocklin https://github.com/mrocklin, see PR that is referenced above.
I check if the numpy array is a numpy array of objects first. In which
case, it is basically equivalent to a dictionary (which may be arbitrary
large as well).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/3692#issuecomment-401567326, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszP1mu1owjM_UBQH7BtT0C82-d2Ahks5uB-2NgaJpZM4U9y-n
.

@mrocklin are you talking about numpy.chararray or simply np.empty((3, 3), dtype=object) that happens to contain strings?
Maybe instead of checking for numpy arrays in the PR I submitted, I can check for dask arrays of objects?

Object arrays
import numpy as np
np.array(['a', 'bb', 'ccc'] * 1000000, dtype=object)
Or, more broadly, we need to make sure that features like this don't affect
performance in other common case workloads.  The example above is one
common case workload that I can think of that I suspect will be harmed.
There may also be others.
On Sat, Jun 30, 2018 at 5:39 PM, Mark Harfouche<EMAIL_ADDRESS>wrote:

@mrocklin https://github.com/mrocklin are you talking about
numpy.chararray
https://docs.scipy.org/doc/numpy/reference/generated/numpy.chararray.html
or simply np.empty((3, 3), dtype=object) that happens to contain strings?
Maybe instead of checking for numpy arrays in the PR I submitted, I can
check for dask arrays of objects?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/3692#issuecomment-401567859, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszK6lczOGQKRW1ixRnczshVCekd3Nks5uB_AEgaJpZM4U9y-n
.

@jakirkham I haven't forgotten about you. I was trying to create a nice self contained example of what I thought your suggestion was. Here is the gist, the array isn't actually computed through the delayed interface: https://gist.github.com/hmaarrfk/3852d40bc82362499f48083092efb702
@mrocklin totally true, that is not a use case I have come across myself.
Maybe the answer is to learn how to use the collections api and create my own array-like collection.

that is not a use case I have come across myself

It's pretty common whenever people deal with text.  For example this data might be names of people in a Pandas series or numpy array.

@mrocklin got it
Seems like I should be able to create a dask collection that does what I want.
I'm trying to go over the documentation again, now that I understand a little more how to use dask.


🐛 fix(config.ts): add support for new OCO_MODEL value 'gpt-3.5-turbo-…

…0613'
The configValidators object now includes the new value 'gpt-3.5-turbo-0613' for the OCO_MODEL key. This allows users to use the latest version of the model. Additionally, the getConfig function now parses the OCO_OPENAI_MAX_TOKENS environment variable as a number if it is defined. Finally, the getConfig function now checks for 'null' and 'undefined' values in the config file and sets them to undefined.

LGTM

i will add this manually now, due to conflicts

thank you <3


Wildcards in tag exclusions

osm2pgsql allows wildcards in the list of tags to drop, e.g. tiger:*.  Based on my reading of mapping/config.go, there's no equivalent way of defining them here, though it would be nice.  Same syntax would be nice (it's concise), but this could also work (borrowing from type_mappings):
{
...
  "exclude": [
    { "tiger": ["__any__"] }
  ]
}

probably related with: https://github.com/omniscale/imposm3/pull/53 , and now solved with this patch https://github.com/omniscale/imposm3/commit/fccaec3977babbe3e8fa063a5071a27563304ab3
see  [documentation] ( http://imposm.org/docs/imposm3/latest/mapping.html )   & search for  "tiger:*"
"tags": {
    "load_all": true,
    "exclude": [
        "created_by",
        "source",
        "tiger:*"
    ]
},


Bug? Coordinate conversion functions yield different results.

Hi there,
Imagine that we have a new OpenSeadragon.Rect, with top-left coordinates 2617, 3600.
We translate that to viewport coordinates, using two methods and comparing the result.
viewport.imageToViewportRectangle(imageRect) gives us a rectangle with top-left coordinates roughly 300, 400
viewport.imageToWindowCoordinates(topLeftPoint) gives us top-left coordinates roughly 308, 408
Any ideas why there would be a difference?  It should be doing the same thing, no?
Thanks in advance.

When I put my code on codepen.io, I changed the styling of the div to use OpenSeadragon's, basically.  So instead of setting a style to the div, with width: 800px and height: 600px, it now uses:
.openseadragon.front-page {
    height: 586px;
}

and then zooming to the ROI and drawing a square around it works.
So.... if people are using stylesheets other than that one... it's not going to work.  I spent weeks trying to figure this out, then by accident I stumbled upon the css being a problem.  May I please have some assistance with this?


x/tools/godoc: godoc doesn't respect the implicit GOROOT / -goroot

What version of Go are you using (go version)?
go version go1.10.1 linux/amd64
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local"
GOTMPDIR=""
GOTOOLDIR="/usr/local/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build709421669=/tmp/go-build -gno-record-gcc-switches"

What did you do?
Use the following Dockerfile:
FROM ubuntu

ADD https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz /tmp
RUN tar -C /usr/local --strip-components=1 -xzf /tmp/go1.10.1.linux-amd64.tar.gz
RUN mkdir -p /go/src/foo
RUN printf 'package main\nimport "fmt"\nfunc main() {fmt.Println("Hello, godoc")}' >/go/src/foo/main.go

ENV GOPATH /go

Please note that this is installing Go directly under /usr/local and not /usr/local/go as the recommended installation instructions do.  I haven't found any issue with this so far other than this problem with godoc.
Build the image with docker build . and then fire it up with docker run -it <the-image-you-built> bash
Once in the container:
root@1015578e9135:/# godoc -http=:8055 -analysis=type
2018/04/23 08:29:32 newDirectory(/): stat /usr/local/go: no such file or directory
2018/04/23 08:29:32 godoc: corpus fstree is nil

root@1015578e9135:/# godoc -goroot=/usr/local -http=:8055 -analysis=type
2018/04/23 08:29:36 GOROOT=/usr/local/go/src/: []
2018/04/23 08:29:36 GOPATH[0]=/go/src/: [foo]
2018/04/23 08:29:36 Loading and type-checking packages...
/go/src/foo/main.go:2:8: could not import fmt (cannot find package "fmt" in any of:
	/usr/local/go/src/fmt (from $GOROOT)
	/go/src/fmt (from $GOPATH))
2018/04/23 08:29:36 Loaded 1 packages.
2018/04/23 08:29:36 Constructing SSA form...
2018/04/23 08:29:36 SSA construction complete
2018/04/23 08:29:36 Computing implements relation...
2018/04/23 08:29:36 Extracting type info...
2018/04/23 08:29:36 Visit instructions...
2018/04/23 08:29:36 Visit instructions complete
2018/04/23 08:29:36 Type analysis complete.

The only thing that works is setting the GOROOT environment variable:
root@1015578e9135:/# GOROOT=/usr/local godoc -http=:8055 -analysis=type
2018/04/23 08:29:49 GOROOT=/usr/local/src/: [archive archive/tar ...etc...]
2018/04/23 08:29:50 GOPATH[0]=/go/src/: [foo]
2018/04/23 08:29:50 Loading and type-checking packages...
[...]

What did you expect to see?
godoc should be able to load the standard library by honoring the default implicit value of GOROOT as printed by go env.  Or at least setting the -goroot flag should work.
What did you see instead?
godoc fails to load standard library packages unless GOROOT is set.

Awww, of course it's right after filing the issue that I find it's a dup.  This is already filed under #23445.  I swear I looked for godoc issues first, but not hard enough apparently!  Sorry for the noise.


Slayer highlight targets tile option

Slayer monsters can be highlighted in the Slayer plugin but only highlighted. I would like a tile highlighter option like the NPC Indicators plugin has currently.

Related to (maybe duplicate of?) #4606. Currently there's no simple way to add this behavior without fully duplicating the code used by the NPC Indicators plugin.

Looks like this is a duplicate. Sorry about that but it would be really nice to have this feature. I really dislike the hull option and the tiles look a lot cleaner and nicer imo


About ImageNet pre-trained weights and ResNet architecture

Hello, thanks for sharing your excellent work!
I have questions about ImageNet pre-trained weights and ResNet architecture.
As you have already mentioned on Getting Started page, you utilized the same checkpoints as provided by the CPS.
Q1. But why did you use privately offered pre-trained weights (by CPS authors) instead of official PyTorch ImageNet resnet pretrained weight? Is there any reason for that?
Q2. Also, have you observed any final performance difference between the two versions of pre-trained weights (by CPS and Pytorch)?
Q3. Why do you use a modified version of the ResNet backbone network? It differs from the original one, AFAIK (you use deep stem resnet by Hang Zhang, not the original one). And you should have mentioned it on the paper... Is it fair to compare your results with other previous work?
Q4. Can you release the performance of your work with the original resnet backbone network (or, to say, torchvision resnet), which utilizes Pytorch-provided ImageNet pretrained weight?
Thanks.

A1. We used the pre-trained weight based on CPS for fair competition, and we also utilise the same sample ID, resnet architecture as CPS.
A2. We didn't compare such difference and the reason is in A1.
A3. Same reason as in A1.
A4. Same reason as in A1.
Please feel free to manage experiments based on the original image-net checkpoint and resnet architecture, but we prefer to follow the released work. Note: all the results in the deeplabv3+ experimental tables are borrowed from CPS, and they are based on "deep stem" if you read the code in here.
By the way, the other peer works utilise more dilation convolutions for the last 2 layers in ResNet which seems can further improve the performance.  I always suggest reading more literature before asking questions.
Cheers

Thanks for your reply! I was just wondering why such work did not mention it (using a modified version of resnet) on their paper, including CPS. Anyway, I got it. Thanks for sharing your work.


Uniswap v3 integration

This week @twalth3r and me started looking at uniswap v3 integration.
The good news is that apparently it is not hard to integrate this pool into our solver, and Tom even developed an experimental PoC.
On the other hand, during the process of collecting some testing instances, I realized that it is not easy at all to read the required pool parameters from the blockchain. Since I'm going on 2 weeks vacation, here's my notes in case someone wants to start on this task.
Each uniswap v3 pool holds a int->int dictionary which we need in the solver (the interpretation is here. There is not, as far as I can tell, a way to get the complete dictionary from the blockchain in one call.
The keys of this dictionary are also in a large bitmap, split in 65536 256-bit segments (words). Given a word, there is a smart contract method to collect all the (dictionary) values corresponding to the keys which are "enabled" in the bitmap (i.e. corresponding bit is one).
The problem is we don't know which words are non-zero, and we don't want to do 65536 smart contract calls to get the dictionary values. We don't even know what is the minimum and maximum non-zero words.
On the other hand, we know one word that is not zero (which corresponds to the "current" price). We also know, that the sum of the values in the dictionary must sum to zero.
To get around this (code here), I implemented an heuristic: we start from this known non-zero word, and then look for non-zero words going outwards from this initial word. We stop looking when the sum of values is zero and we don't find a new non-zero word in the most recent N attemps.
Anyway, the heuristic is not so important. The fact is that even after chatting with the uniswap team (see the thread in our dedicated slack channel) it appears that there is no way to get what we need efficiently. Even this heuristic takes 20 seconds to download dictionary of the largest pool (in TVL) using infura.
So I suppose we need to maintain some internal database mirror of the pools and keep it synchronized via events or something, so that data is readily available. I still think that a special purpose service for all our pools that does this task would make a lot of sense (like the orderbook but for liquidity).

So I suppose we need to maintain some internal database mirror of the pools and keep it synchronized via events or something, so that data is readily available. I still think that a special purpose service for all our pools that does this task would make a lot of sense.

This really screams "Let's use TheGraph for this" to me.
Have you already had the chance to tinker with some of the existing uniswap subgraph implementations? We could also ask Paraswap/Matcha how they index the Uniswap v3 state.

Yes, I also tried uniswap v3 subgraph, but I suspect it has a bug. Nonetheless, if it was working correctly, and perhaps if we could use a self hosted version, maybe that would work (my experience with the public thegraph is that it is rather slow).


Unable to use WHERE clause to filter by tags in Influx http api

Hi Folks,
Firstly apologies if this is a silly question, but I can't get this working at all.


fix(nuxt): useCookie regression

🔗 Linked issue
https://github.com/nuxt/nuxt/issues/25587
❓ Type of change

[ ] 📖 Documentation (updates to the documentation, readme or JSdoc annotations)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
Fixes the regression introduced by https://github.com/nuxt/nuxt/pull/25198.
Improved tests to also test if value changes across useCookie invocations.
📝 Checklist

[x] I have linked an issue or discussion.
[x] I have added tests (if possible).
[ ] I have updated the documentation accordingly.

Thank you very much ❤


feat: Auto delete old charts

Closes #370
Output when importing zips

Nice, I'm glad you're tackling this! Once concern that I tried to raise in the original ticket is that we can't assume that any assets in Superset that aren't current Aspects assets should be deleted (like custom charts and dashboards) so we need some kind of way to say "this asset uuid is a deleted Aspects asset from the past that should be cleaned up".

Nice, I'm glad you're tackling this! Once concern that I tried to raise in the original ticket is that we can't assume that any assets in Superset that aren't current Aspects assets should be deleted (like custom charts and dashboards) so we need some kind of way to say "this asset uuid is a deleted Aspects asset from the past that should be cleaned up".

could we instead have a file that keeps a list of unused assets that we DONT want to delete? that might be easier to maintain than a list of ones TO delete

I don't think we have a way of knowing which custom assets someone added are unused / deletable. We can only know our own assets, so we could put a check in CI or something that would fail a build if there were unused assets left in our changes. Then we could add them to the list. Seems kind of error prone, but if we put the reminder to add them to the list in the error message it might help?

Ahhh makes sense. Cool, I'll do that


[Feature request] NOSTR relay "allow list" switch

Description
A simple switch button to activate "allow list only" to join to the relay.
Right now, in the relay config user can activate the paid option and in that moment only allowed / paid pubkeys can join.
But what about running a simple relay for family / friends for free but in a private mode, only predetermined allowed pubkeys?
Yes, we can put a high fee and add a bunch of pubkeys as allowed, but nowadays not many people are so smart to take advantage of 1BTC fee for a nostr relay 😂😂😂.
Solution proposal

So a more visible button in the accounts section will be much easier for users that do not want to use it with the payment method.
Adding a bunch of allowed pubkeys and switch the button "Allow list - block all the rest".
If the button is not activated, then anybody could join, but ONLY if the payment option is disabled.

is this feature referencing NIP-29?
@motorina0 , is @arcbtc  still working on this issue? would be nice to have. I would like it.
https://github.com/nostr-protocol/nips/blob/fade0164f52033314bf0a5ef9bd63c2483afae9b/29.md

@arcbtc please assign to me.
cc: @Darth-Coin

OK now this repo is updated to be inline with V1.0, i will start to implement the feature. @dni

go for it :)

I am still working on it. been too many things and not enough time. If i free up my plate this week, eta is mid december now


Option to export trained models as ONNX

So far this code works but shows some warnings, I'm not sure if these can be ignored or should be resolved
Resolves #23

We can definitely merge the PR when it is finished. I tried it out in Godot 3.5.1 mono and it was not clear how to load the model file. Is this still WIP?

Yes, right now it just loads the model, but does nothing with it, I have yet to implement that part.
There is a scene named "Test.tscn" If you run that one you'll load the model and create an inference session.

Doing some clean up, got the model to run and produce an output

This is really cool, I tried out the PR on my machine, but I am on Linux and I don't think I have the right ONNX runtime. I am not so familiar with using C# in Godot. Do you have any advice about installing the dependencies?

@edbeeching I just pushed some instructions in the "Installation" docs

Most recent change, now it detects and uses any GPU it finds for the Inference Session. If it's not compatible, it just uses the CPU.

I took me a moment to get it working on Linux (Ubuntu 20.04), in the end I only got inference working on CPU.
I had to manually add the lib to the .csproj file with:
  <ItemGroup>
   <None Include="libonnxruntime.so" />
   <ContentWithTargetPath Include="libonnxruntime.so">
     <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     <TargetPath>libonnxruntime.so</TargetPath>
   </ContentWithTargetPath>
 </ItemGroup>

In SessionConfigurator.s I had to comment out line 51:
//options.AppendExecutionProvider_CUDA(0);

Otherwise, I have the following error:
E 0:00:00.274   System.EntryPointNotFoundException: OrtSessionOptionsAppendExecutionProvider_CUDA assembly:<unknown assembly> type:<unknown type> member:(null)
  <C++ Error>   Unhandled exception
  <C++ Source>  :0
  <Stack Trace> :0 @ ()
                :0 @ void Microsoft.ML.OnnxRuntime.SessionOptions.AppendExecutionProvider_CUDA(Int32 )()
                SessionConfigurator.cs:51 @ Int32 GodotONNX.SessionConfigurator.SystemCheck()()
                ONNXInference.cs:22 @ void GodotONNX.ONNXInference.Initialize(System.String , Int32 )()

This may be because I have a RTX 4090 and the OnnxRuntime does not support this yet?
Regarding the model, is this a trained policy? I will try and get it running the agent over the holiday.

Did you try the fix I pushed? changed the csproj to include a target platform, that got the program to run.
It's not the GPU, seems that as I'm trying to let the SessionConfigurator.cs pick which execution provider to use, there's a bit of ambiguiety that prevents it from loading the correct library. I'm currently working on this.
For the exported models, these are the trained checkpoints available as examples, restored and then converted to ONNX, so you can find a model.onnx inside each checkpoint folder

I've been searching for an answer on this particular issue, from getting all dependencies (seems you need to install CUDA 11.6 with cuDNN 8.5.0 [windows] or 8.2.4 [Linux] too) to trying to manually assign the dll. So far the best I could get was to comment the CUDA line as well.
DirectML works fine, either on AMD or NVDIA.

As I see the repo has moved to 4.x support, I think this should be merged into a 3.X branch, from there, I'd backport features from 4.X to 3.X and update parts of this PR into a new PR for 4.X

Hey, sorry for the slow reply. I was off for the holidays.
I have created a branch called godot3.5, if you can update the PR to be on that branch we can look to merge this week. Going forward, would you like to have collaborator access to this repo, so you can make your changes directly on branches here?

No worries! I hope you had happy holidays.
I made the necessary changes to the PR, and I'd appreciate a lot the access!
I would still create some issues or PRs for discussion if needed.

@edbeeching is it ok if I merge this? As the next step for the 3.5 branch is backporting the python code, some changes here might get overriden, but that's ok, so I could procede to setup the export process for the current python code

Hey, sure. just fix the merge conflict in .gitignore and it is good to go.

Merged! 🎉

@yaelatletl this looks cool! Will it work in the end like in unity ml agents so that you can just plug in the onnx file into the agent?

Nice, there seems to be some duplication of files (ONNXInference and ONNXWrapper) inside JumperHard and the addon directory. I deleted the ones not in the addon dir to get the Test scene to build. I now have an exception related to SessionConfigurator.SystemCheck();, I will try and look into this more tonight.
On another note, would it be interesting to discuss at some point what features you think are missing from this lib? Perhaps I / we can implement them next.

@edbeeching I just scrolled through this PR and read the discussion. Regarding the question what could be improved, a few things come into my mind which have the potential to improve the whole thing a lot.

Provide a detailed and clean documentation, on how to use things. I think that's one of the big features which decides if people will use a tool or not. Make it as easy as possible for new people to get started. Especially since RL is already hard enough. Currently the framework is still a lot of trial and error and some things only work on certain platforms without being documented.
Provide a step by step video (+ text) tutorial on how to setup and train an agnet with godo rl agents from scratch. I think this will help newcomers immensely to get started.
Make sure the provided releases actually work. It took me quiet some time until I realized that the senord3d is not working in Releaes 0.2.2. I guess it makes sense to create a new release and remove the broken ones.
It would be cool, if we GDScript would provide interfaces so it gets really clear which methods and members the agent has to implement. I know GDScript does not support this, but maybe there is an alternative?

Cheers, hope this helps!

@visuallization , thanks for highlighting this. I am aware of the poor quality of the docs. I have been focused on features and my time is split across a number of projects. For the Hugging Face Deep RL course, I am making a more detailed tutorial, which will be released in a couple of weeks. I will aim to add that to the docs as well.
I have created an issue #59 to track these improvements for the next version.

@yaelatletl this looks cool! Will it work in the end like in unity ml agents so that you can just plug in the onnx file into the agent?

Hey, thanks! Yes, that's the plan, just loading the exported file and be good to go.

@yaelatletl
Do you have a working exported Onnx and working within Godot-RL Agent?


Hold multitouch on TouchScreenButton

Godot version:
3.1 - 3.2.dev.custom_build
OS/device including version:
Android 7 (nougat)
Issue description:
When i make 3 or more touch in same time (one button = one touch) and hold for 3 or more sec, all buttons is auto released even my fingers still touching the screen. It's not happen when i make 2 or less touch.
Issue gone when i make 1 touch (any position on screen) first (hold), after that holding 3 or more touch in same time is work.
Steps to reproduce:

Add 5 or more TouchScreenButton
Use different image on normal & pressed (to know when button is pressed or released)
Export to android
Make 3 or more touch (in same time) on different button (on finger one button)

Which phone model is it exactly?
I have the suspicion that the three-finger gesture is being handled by the OS (for screen capture or something) and that is interfering somehow.
Just an idea.

Which phone model is it exactly?
I have the suspicion that the three-finger gesture is being handled by the OS (for screen capture or something) and that is interfering somehow.
Just an idea.

You right, i just test (disable 3 finger screenshot) and issue gone.. Thanks
I close this..


RuntimeException: Waited 5000ms for main-FPSAWTAnimator

I'm attempting to run your code on Processing 3.3.7, I'm using the latest version of the libraries you included that I could find but I'm not able to get passed this error. The full error text is:
java.lang.RuntimeException: Waited 5000ms for: <6e53978c, 60947436>[count 2, qsz 0, owner <main-FPSAWTAnimator#00-Timer0>] - <main-FPSAWTAnimator#00-Timer0-FPSAWTAnimator#00-Timer1> 	at processing.opengl.PSurfaceJOGL$2.run(PSurfaceJOGL.java:410) 	at java.lang.Thread.run(Thread.java:748)
I assume its an error with opengl since they made updates to how P2D works, but I can't seem to find any solutions online.

So I got it working. I had to move the Wifi code from the loop to the setup, as it was trying to connect again even when it was connected for some reason. Then I had to update the data/config.xml file to match my local IP on the device (this wasnt evident as there is no real readme or instructions). Thank you @hubmartin this is amazing.

I'm glad you have it running. Can you post a video of animation on the Dotstar? I never used this streaming on any 2D surface, I'd be interested in the framerate and stability. Thanks.

I can post a quick video that I took, but I need to reduce the brightness and take another video. Also, what do you mean you've never used this streaming on any 2D surface? Link to GIF
The stability isn't bad but there seems to be a bit of lag when I move the LEDs around the canvas in processing.

Thanks for video in action. Looks very fast. I had some issues with 5 or more of these devices on my router. Not sure what was the cause.


Components to modify dl1 images

As recently discussed (See #1706, #1707 and linked lstchain PR), modifying simulated dl1 images is a useful way to improve data/mc mismatches.
There are different ways to achieve this. I personally feel like it is easiest done in stage-1, although that requires to modify the event loop. Currently I added a ImageModifier component, which forced me to also add a NullModifier, that does not alter the image. There might be better ways to achieve this in the given traitlets system.
While I feel like this is somewhat hacky, we might want to use this to mask pixels or do other things in real data, so that this would not be used all the time in the end anyway (?).
The advantage of implementing it this way to me is that we already have a tool, that can calculate dl1 from input dl1 files, which is also used for recleaning anyway and also takes care of recalculating parameters. This makes it easy to use from a users perspective and less duplicated effort.
This is certainly a point to be discussed though.
As there are countless ways to modify images and being able to reproduce lstchain results is a plus anyway, I for now implemented the approach @moralejo used in lstchain as a ready to use component. This performs the smearing aswell as the addition of noise.

Opinions on using stage-1? Is there any benefit in a new tool?
Are there better ways than using a dummy modifier? Does it matter?
It should be easy to perform this on all images at once, just like the cleaning. I think its not worth to do this here though, especially since this would be part of the image processing loop, that could be changed altogether.

Proof of concept: I processed some simtel events and reprocessed the dl1 file with stage-1, using the LSTImageModifier.

Wouldn't be easier to generalize ctapipe.image.toymodel.ImageModel and/or what's in that sub-module instead of adding new functions?

I think having basic functions is better tbh. It's easier to test this way and if we want to avoid duplicate code we could add this to the toymodel (which I havent done yet).
And in my understanding the toymodel is not meant as a tool to use in analyses, but more of a visualization/testing thing. Correct me if I am wrong.

To be clear I am not criticizing what has been added here, just wondering if it makes sense how much code makes sense to replicate wrt what we already have

I think having basic functions is better tbh. It's easier to test this way and if we want to avoid duplicate code we could add this to the toymodel (which I havent done yet).

add_noise is basically the same as ImageModel.generate_image so you could just use/modify/generalize that no?
I would say it' better to have all these functions in the same place and let the ImageModifier pick what it needs

And in my understanding the toymodel is not meant as a tool to use in analyses, but more of a visualization/testing thing. Correct me if I am wrong.

There the signal is created randomly, but nothing prevents you (us) to add a keyword argument image=Nonewhich if initialized with dl1.image uses that instead of creating the randomized toymodel image

@HealthyPear As noted in the connected issue, I think using the ImageModel in ModifyImage is having it backwards.
It should be the other way around. The ImageModel should use the ImageModifier to add noise / smear the image.

I have a few design questions/comments:

How is this expected to be included in the main analysis (e.g. ctapipe-stage1): a single step in the chain?
Is the idea that there can be one and only one ImageModifier that can be chosen by name? what if the user wants to apply two or more modifications?
would it be be better to drop the hierarchy and have one big one that has many options (e.g. to be able to smear and add noise at the same time in a generic way?) Or do you expect that each chain would build their own subclass and enable that? (meaning code has to be written to support fancier combinations of modifications?)

For the question of where to put it: I think it should go in ctapipe-stage1, but also be a separate tool. That way one can use it either way during analysis, or on an existing DL1 file to generate a new modified file.   That bring up the question of if it should act event-wise or on all events at once (which would be more efficient in the second case)

For the question of where to put it: I think it should go in ctapipe-stage1, but also be a separate tool. That way one can use it either way during analysis, or on an existing DL1 file to generate a new modified file.

For that you don't need a separate tool, you just rerun stage1 using the dl1 source.

* How is this expected to be included in the main analysis (e.g. ctapipe-stage1): a single step in the chain?

Thats my take. What alternatives do you see?

* Is the idea that there can be one and only one ImageModifier that can be chosen by name? what if the user wants to apply two or more modifications?
* would it be be better to drop the hierarchy and have one big one that has many options (e.g. to be able to smear and add noise at the same time in a generic way?) Or do you expect that each chain would build their own subclass and enable that? (meaning code has to be written to support fancier combinations of modifications?)

This is also one of my main concerns. Unfortunately I dont really see a way out. Building one tool with lots of options did not strike me as more elegant though. If you or someone else sees a way out, I would be very interested. Afaik there is no easy way to enable multiple (or an a priori undefined number of) components of the same type in a tool, so we would need a monolithic giant at some point. We could also have one component for each task (adding noise, blurring the light, masking pixels, ...). I settled with one component, but I dont have a hard opinion.

That brings up the question of if it should act event-wise or on all events at once (which would be more efficient in the second case)

Agreed. I feel like it could be done on all events just like the cleaning could. But since you would need to reclean and recalculate parameters anyway, I figured its not worth the effort to do it independently.

For that you don't need a separate tool, you just rerun stage1 using the dl1 source

Good point. It's not super efficient, but should work.

It does work and is also done for recleaning dl1 files without modifying the images.
But having the event loop is of course not optimal in terms of runtime.

It does work and is also done for recleaning dl1 files without modifying the images.
But having the event loop is of course not optimal in terms of runtime.

Yeah, let's ignore speed issues for now - that can be a future refactoring of ctapipe in general. The design now is also fine, having a single component with subclasses like the ImageExtractors.  It's not super flexible, but that's ok for now, and I can't think of a better way : even with one big monster class, the order of operations is hard to define (e.g. blur then add noise is not the same thing as add noise then blur).

@LukasNickel  did you see the updated function for the PSF here: https://github.com/cta-observatory/cta-lstchain/pull/719 ?
The first version had some problems discussed in that PR

@LukasNickel did you see the updated function for the PSF here: cta-observatory/cta-lstchain#719 ?
The first version had some problems discussed in that PR

I did, but would have probably forgotten to incorporate it, thanks

Relying on #1776.
Otherwise this doesnt work on dl1 files, that have parameters and images

Matplotlib errors in seemingly unrelated code only in the push test?
Any idea where that is coming from? I already reran the tests...

@LukasNickel You need to merge or rebase master, the issue is fixed there. Since the pull request build runs on the merged version, it passes.

@LukasNickel could you implement the random seed stuff we talked about?
I.e. add a seed configurable to the Component and generate seeds from that for the numba code?


Need for Vuejs Please

good news, we are working on the v1 release, it will be framework agnostic and there will be an adapter for Vuejs


Custom dataset with "from_generator"

Hi!
First off, thanks for the great work with a TF2 compatible DETR code!
I've been working with object detection using EfficientDet and my own dataset. The dataset consists of synthetic data, from a generator function, something along the lines of
def generator():

    while True:

        # Do work

        # yields the image, the label, and the bounding box
        yield (image, ([label], [[x1, y1, x2, y2]]))

My initial tensorflow dataset object is then made from:
output_shapes = (tf.TensorShape([512,512, 3]),
                    (tf.TensorShape([None]),
                    tf.TensorShape([None, 4])))

ds = (tf.data.Dataset
        .from_generator(generator=generator,
                        output_types=(tf.float32, (tf.int32, tf.float32)),
                        output_shapes=output_shapes)
        )

However, the output of the tensorflow_datasets imports is in the form a dict.
Any way we can get a custom generator to work with your current code?
Thanks!!

Hi,
You don't need much.
I guess something like that would work.
from kerod.core.standard_fields import BoxField, DatasetField

def transform(image, gt):
    inputs = {
        DatasetField.IMAGES: image,
        DatasetField.IMAGES_INFO: tf.constant((512, 512, 3))
    }

    ground_truths = {
        BoxField.BOXES: gt[0],
        BoxField.LABELS: gt[1],
        BoxField.NUM_BOXES: tf.shape(gt[0]),
        BoxField.WEIGHTS: tf.fill(tf.shape(gt[0]), 1.0)
    }
    return inputs, ground_truths

output_shapes = (tf.TensorShape([512,512, 3]),
                    (tf.TensorShape([None]),
                    tf.TensorShape([None, 4])))

ds = (tf.data.Dataset
        .from_generator(generator=generator,
                        output_types=(tf.float32, (tf.int32, tf.float32)),
                        output_shapes=output_shapes)
        )

ds = ds.map(transform, num_parallel_calls=tf.data.experimental.AUTOTUNE)

Don't forget to batch everything after. Look at the notebooks for batching example.

Hi,
Thanks! I think the solution is almost there, but there's a small error I can't pin down. I have made a very small dummy generator, to highlight what's happening:
import numpy as np
import tensorflow as tf
from kerod.core.standard_fields import BoxField, DatasetField

batch_size = 8

def generator():

    while True:
        image = np.zeros((512,512,3))
        label = 1

        x1 = 100.
        x2 = 300.
        y1 = 100.
        y2 = 300.

        yield (image, ([label], [[x1, y1, x2, y2]]))

def transform(image, gt):
    inputs = {
        DatasetField.IMAGES: image,
        DatasetField.IMAGES_INFO: tf.constant((512, 512, 3)),
        DatasetField.IMAGES_PMASK: tf.ones((512, 512))
    }

    ground_truths = {
        BoxField.BOXES: gt[1],
        BoxField.LABELS: gt[0],
        BoxField.NUM_BOXES: tf.shape(gt[1]),
        BoxField.WEIGHTS: tf.fill(tf.shape(gt[1]), 1.0)
    }
    return inputs, ground_truths

output_shapes = (tf.TensorShape([512,512, 3]),
                    (tf.TensorShape([None]),
                    tf.TensorShape([None, 4])))

ds = (tf.data.Dataset
        .from_generator(generator=generator,
                        output_types=(tf.float32, (tf.int32, tf.float32)),
                        output_shapes=output_shapes)
        )

for img, (label, bbs) in ds:
    break

print(img.shape)
print(label.shape)
print(bbs.shape)

So far so good - the output of this looks good, in terms of shapes returned:
(512, 512, 3)
(1,)
(1, 4)

Now, running the transformation you provided, the outputs again looks as they should (actually, they look identical to the outputs of your notebooks, as far as I can tell):
ds = ds.map(transform, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for datapoint in ds:
    break
print(datapoint[0]['images'].shape)
print(datapoint[1]['bbox'])
print(list(datapoint[0].keys()))
print(list(datapoint[1].keys()))

With output:
(512, 512, 3)
tf.Tensor([[100. 100. 300. 300.]], shape=(1, 4), dtype=float32)
['images', 'images_information', 'images_padding_mask']
['bbox', 'label', 'num_boxes', 'weights']

It looks like all keys and shapes are correct!
Finally, when moving to the padded_batch, I get errors:
padded_shape = ({
  DatasetField.IMAGES: [None, None, 3],
  DatasetField.IMAGES_INFO: [2],
  DatasetField.IMAGES_PMASK: [None, None]
},
{
  BoxField.BOXES: [None, 4],
  BoxField.LABELS: [None],
  BoxField.NUM_BOXES: [1],
  BoxField.WEIGHTS: [None]
})

# Filter example with no boxes after preprocessing
ds_train =  ds.padded_batch(batch_size, padded_shape)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-d3eee6882481> in <module>()
     76
     77 # Filter example with no boxes after preprocessing
---> 78 ds_train =  ds.padded_batch(batch_size, padded_shape)
     79
     80

2 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/data/ops/dataset_ops.py in _padded_shape_to_tensor(padded_shape, input_component_shape)
   4042     raise ValueError("The padded shape %s is not compatible with the "
   4043                      "corresponding input component shape %s."
-> 4044                      % (padded_shape_as_shape, input_component_shape))
   4045
   4046   return ret

ValueError: The padded shape (2,) is not compatible with the corresponding input component shape (3,).

This is probably a simple mistake, but I don't see why it thinks the padded shape should be 2...
Thanks!

DatasetField.IMAGES_INFO: tf.constant((512, 512))
}

Thank you! Making progress...
The following code executes without trouble, and as far as I can see the Dataset yields exactly the same type of data as the one loaded from tfds.
import numpy as np
import tensorflow as tf
from kerod.core.standard_fields import BoxField, DatasetField

batch_size = 2

imgSize_x = 512
imgSize_y = 512

def generator():

    while True:
        image = np.random.rand(imgSize_x, imgSize_y,3)
        label = 1

        x1 = 100.
        x2 = 300.
        y1 = 100.
        y2 = 300.

        yield (image, ([label], [[x1, y1, x2, y2]], np.ones((imgSize_x, imgSize_y), dtype=np.int8)))

output_shapes = (tf.TensorShape([imgSize_x, imgSize_y, 3]),
                    (tf.TensorShape([None]),
                    tf.TensorShape([None, 4]),
                     tf.TensorShape([imgSize_x, imgSize_y])))

ds = (tf.data.Dataset
        .from_generator(generator=generator,
                        output_types=(tf.float32, (tf.int32, tf.float32, tf.int8)),
                        output_shapes=output_shapes)
        )

def transform(image, gt):
    inputs = {
        DatasetField.IMAGES: image,
        DatasetField.IMAGES_INFO: tf.constant((imgSize_x, imgSize_y)),
        DatasetField.IMAGES_PMASK: gt[2]
    }

    ground_truths = {
        BoxField.BOXES: gt[1],
        BoxField.LABELS: gt[0],
        BoxField.NUM_BOXES: tf.shape(gt[1]),
        BoxField.WEIGHTS: tf.fill(tf.shape(gt[1]), 1.0)
    }
    return inputs, ground_truths

ds = ds.map(transform, num_parallel_calls=tf.data.experimental.AUTOTUNE).repeat(-1)

padded_shape = ({
  DatasetField.IMAGES: [None, None, 3],
  DatasetField.IMAGES_INFO: [2],
  DatasetField.IMAGES_PMASK: [None, None]
},
{
  BoxField.BOXES: [None, 4],
  BoxField.LABELS: [None],
  BoxField.NUM_BOXES: [None],
  BoxField.WEIGHTS: [None,4]
})

ds_trainSynth =  ds.padded_batch(batch_size, padded_shape, drop_remainder=True)

However, upon training, the following error occours:
history = model.fit(ds_trainSynth,
                    epochs=50,
                    steps_per_epoch=512)

Epoch 1/50
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
WARNING:tensorflow:Using a while_loop for converting EagerPyFunc
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-62-9184344663df> in <module>()
      1 history = model.fit(ds_trainSynth,
      2                     epochs=50,
----> 3                     steps_per_epoch=512)

6 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Can not squeeze dim[1], expected a dimension of 1, got 2
	 [[node Squeeze_5 (defined at /usr/local/lib/python3.7/dist-packages/kerod/core/matcher.py:187) ]]
	 [[giou_loss_3/weighted_loss/Equal/_1338]]
  (1) Invalid argument:  Can not squeeze dim[1], expected a dimension of 1, got 2
	 [[node Squeeze_5 (defined at /usr/local/lib/python3.7/dist-packages/kerod/core/matcher.py:187) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_619113]

Errors may have originated from an input operation.
Input Source operations connected to node Squeeze_5:
 IteratorGetNext (defined at <ipython-input-62-9184344663df>:3)

Input Source operations connected to node Squeeze_5:
 IteratorGetNext (defined at <ipython-input-62-9184344663df>:3)

Function call stack:
train_function -> train_function


Problem with sequential model training

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.
Thank you!

[v ] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

[ v] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[ ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

I'm trying to train sequential model.
I defined two model like this
def model1 :
inputs = Input(input_shape=(256,256,3))
output = Conv2d(channels = ch, output=(w,h) , name='model1_conv')(inputs)
model1 = Model(inputs = [inputs], outputs= output)
def real_model:
input_tensor = Input(input_shape=(256,256,3))
F1 = model1(inputs=(256,256,3))
fm = Model(inputs=F1.input, outputs=F1.output)
fm_out1 = fm(data1)
fm_out2 = fm(data2)
conc_out = concate(fm_out1,fm_out2)
conc_out = conv2d(name='real_conv')(conc_out)
conc_out = sigmoid()(conc_out)
model = Model(inputs=input_tensor, outputs= conc_out)
and the results seems that only the conv2d layer in real_model ('real_conv') is trainied and model1 weights  is not changing at all.
I don't know how to solve this. Is there any one who can solve this ?
Thank for reading.

Just a hunch but
i found these two things to be bugging:

fm = Model(inputs=F1.input, outputs=F1.output) you have given F1.input and F1.output instead of F1.inputs and F1.outputs, maybe i'm wrong with this.
concate instead of concatenate in the line conc_out = concate(fm_out1,fm_out2)

@greed2411 Thanks for reply.  above one is pseudo. so real implementation I'm using  written those functions you wrote. I should have mentioned it , sorry.


Consider ifunc symbols also defined

Looks like php may be built with certain optimizations causing some symbols to be resolved as ifunc. The object crate is not recognizing these as definition, but for our purposes, it is one.
Fixes part of Datadog/dd-trace-php#2831.

Codecov Report
Attention: Patch coverage is 10.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 72.67%. Comparing base (e899c91) to head (7c0de1d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #609      +/-   ##
==========================================
- Coverage   72.73%   72.67%   -0.06%
==========================================
  Files         246      246
  Lines       34940    34948       +8
==========================================
- Hits        25412    25398      -14
- Misses       9528     9550      +22

Components
Coverage Δ

crashtracker
20.44% <ø> (ø)

datadog-alloc
98.73% <ø> (ø)

data-pipeline
90.12% <ø> (ø)

data-pipeline-ffi
0.00% <ø> (ø)

ddcommon
82.82% <ø> (ø)

ddcommon-ffi
68.11% <ø> (ø)

ddtelemetry
59.02% <ø> (ø)

ipc
83.61% <ø> (ø)

profiling
84.26% <ø> (ø)

profiling-ffi
77.42% <ø> (ø)

serverless
0.00% <ø> (ø)

sidecar
40.13% <ø> (ø)

sidecar-ffi
0.00% <ø> (ø)

spawn-worker
50.36% <ø> (ø)

tinybytes
91.66% <ø> (ø)

trace-mini-agent
70.88% <ø> (ø)

trace-normalization
98.25% <ø> (ø)

trace-obfuscation
95.73% <ø> (ø)

trace-protobuf
77.67% <ø> (ø)

trace-utils
93.42% <ø> (ø)


pkg(go-1.20.1): Add Go 1.20.1

user@work ~/s/piston (update-golang)> ./piston run go -l 1.20.1 packages/go/1.20.1/test.*
== Run ==
STDOUT
OK

Apparently the test failed due to restraints of its environment:
Running packages/go/1.20.1/test.go with runtime=go
== Run ==
STDERR
# math
    runtime: failed to create new OS thread (have 3 already; errno=11)
    runtime: may need to increase max user processes (ulimit -u)
    fatal error: newosproc

Since test.go basically only contains "hello, world", would it make sense to rerun the test? Or do I need to change something?
package main

import "fmt"

func main() {
	fmt.Println("OK")
}

@HexF anything I can or should do?

@HexF anything I can or should do?

Might need to up the run timeout for tests to pass


Pagination functionality.

Check this out. (Haven't tested, maybe needs a fix)
My recommendations its to remove all $perPage, references. clean all that and just just the new function to read directly from request this 2 vars: limit + page.
What I have seen in other libraries its people gives options to set this vars names, I don't think its necessary, you can always overwrite them.
Its up to you, my suggestions:

Clean, remove all $paginate & $perPage
move functions specs to the contract and just use the,  {@inheritdoc}
show the contract in the readme for people references.

Simple the better.

You need to limit the database query too. So your $query->get() approach is wrong. Also I think people can pass in the item limit from controller easily. No need to overkill a simple thing.
    public function index(Request $request) {
        $this->repo->paginate($request->limit);
    }

You approach also restricts a users ability to use limit as a request parameter while using pagination.

Humm...
$this->repo->paginate() its Laravel defaults.
This pagination implementation needs to know how many records are.
People can do $request->limit = 'xXX'
There is a simplePagination that does not require to know that.
Works well on Lumen, where defaults pagination was not giving right links.
By reading request, we don't need to pass by this vars, but you are the boss here!😋

We can work this out for lumen for sure, it just needs to be cleaned up and go the correct place. We might make a class for lumen or a trait for lumen methods.

Laravels magic resolves the page parameter from the $request, same we doing with limit.
The trick we adding here its to be able to turn pagination on-off by sending the limit in the request.
What this function does its the same than Laravel, but also returns others parameters in the query for the next_page_url and so..
make a test, for example with simple url , check what happens with rest of parameters when you do the paginate using laravel defaults.
Also, have a look at this:: https://github.com/rlacerda83/lumen-api-query-parser/wiki/Usage
its very interesting how parses urls and returns a query build.
with a url like this:
http://musicboxx.dev/v1/role?page=1&order=33&limit=3
Laravel returns:
    "first_page_url": "http://musicboxx.dev/v1/role?page=1",
    "from": 1,
    "last_page": 2,
    "last_page_url": "http://musicboxx.dev/v1/role?page=2",
    "next_page_url": "http://musicboxx.dev/v1/role?page=2",
    "path": "http://musicboxx.dev/v1/role",
    "per_page": 15,
    "prev_page_url": null,
    "to": 15,
    "total": 22

but I think this is more convenient:
    "first_page_url": "http://musicboxx.dev/v1/role?page=1&order=33&limit=3",
    "from": 1,
    "last_page": 8,
    "last_page_url": "http://musicboxx.dev/v1/role?page=8&order=33&limit=3",
    "next_page_url": "http://musicboxx.dev/v1/role?page=2&order=33&limit=3",
    "path": "http://musicboxx.dev/v1/role",
    "per_page": "3",
    "prev_page_url": null,
    "to": 3,
    "total": 22

i would suggest to not "bind" this to specific request parameter (e.g., ?limit=xxx or ?page=xxx) as this would result in not being able to use this package with specific standards (e.g., json:api) that define other parameters..
I think, these query parameters should be customizable in the config file!


Support skipLibCheck in tsconfig

Typescript 2.0 adds the boolean flag skipLibCheck. atom-typescript considers this an invalid option and fails to parse my tsconfig.

Shouldn't these errors just be a warning instead of stopping the whole plugin from working?

👍


Unset Tactician Neural Truncate (default)

very minor and quick (but changes the API so some coordination is required).
As we have full support for update new definitions and typically working on workflows with full graphs, let's switch to the default where Truncate is off, it saves one line for clients playing with the system in the regime its intended ?

This option no longer exists. Closing.


add first test to common pkg

start tests from epic Add unit tests to Edge API.

From reaching out to a bonfire engineer I believe its an issue with that tool.
Could you try rebasing and see if that helps?


ci: update install-snapshot script to deploy newer version of snapshotter

Updated the install-snapshot.sh  to support installing of new snapshotter version and also to enable volumegroupsnapshot flag during installation. Used kubectl_retry helper to retry if there is any failure.

/test ci/centos/k8s-e2e-external-storage/1.27

@mergifyio queue


Add param to fast.urlfilter to filter based on length of the URL

see https://github.com/commoncrawl/ia-web-commons/issues/32
We should already have the conf for it
urlfilter.fast.url.path.max.length1024
urlfilter.fast.url.pathquery.max.length2048
This needs to be done in Nutch and merged back

See https://www.baeldung.com/cs/max-url-length

https://issues.apache.org/jira/browse/NUTCH-3025

https://github.com/apache/nutch/pull/796

The PR uses urlfilter.fast.url.query.max.length instead of  urlfilter.fast.url.pathquery.max.length

Assigned to @sebastian-nagel to review the PR in Nutch land, merge if appropriate then merge back into our cc-nutch repo


Very long build time

Hi @AngeloAvv
Application build time is pushed up very high. Initially, my app took less than 20s to build the first time and only 15s for the second. However, after using flavor it took 150s the first time and 60s the second time.
I know that your flavor library already generates scripts to copy the config files before runtime. However, I don't think that makes the build time so long.
There may be some problems in my project but hope to get your answer to this question.

I remembered that while creating the flavor I added firebase to the app. maybe that makes my app slow to build

Hello @holland-hng , I don't think flutter_flavorizr has anything to do with the building time. Of course the more you add dependencies, the longer the app will take to build

@AngeloAvv
Thanks for your respond
This problem occurs when I integrate firebase ios sdk into the project, it's really too heavy.


How to invoke the panTo() function?

It appears that the panTo function of the Google Maps API is exposed. How do I invoke it in an example?

const MapPanToClick = () => {
  const [position, setPosition] = useState()

  const mapRef = useRef()

  useEffect(() => {
    if (mapRef.current?.panTo && position)
      mapRef.current?.panTo(position)

  }, [position])

  return (
    <GoogleMap
      defaultZoom={15}
      center={position || { lat: -34.397, lng: 150.644 }}
      onClick={e => setPosition({ lat: e.latLng.lat(), lng: e.latLng.lng() })}
      ref={mapRef}
    >
      <Marker position={position} />
    </GoogleMap>
  )
})

@crobinson42
The repo of this project is unmaintained more than a year, and we had build new version https://www.npmjs.com/package/@react-google-maps/api
We had rewrite it to TypeScript, and updating it frequently: https://github.com/JustFly1984/react-google-maps-api/tree/master/packages/react-google-maps-api
You can enjoy autocomplete.
You can see our docs: https://react-google-maps-api-docs.netlify.com/
Also a lot of examples: https://react-google-maps-api-gatsby-demo.netlify.com/ https://github.com/JustFly1984/react-google-maps-api/tree/master/packages/react-google-maps-api-gatsby-example/src/examples
The bundle size is much smaller: https://bundlephobia.com/result?p=@react-google-maps/api
Our Spectrum community: https://spectrum.chat/react-google-maps
Our Slack channel: https://join.slack.com/t/react-google-maps-api/shared_invite/enQtODc5ODU1NTY5MzQ4LTBiNTYzZmY1YmVjYzJhZThkMGU0YzUwZjJkNGJmYjk4YjQyYjZhMDk2YThlZGEzNDc0M2RhNjBmMWE4ZTJiMjQ
Enjoy!


Duplicate react-native module error!

Package is installing another version of react-native which is conflicting.
Fix for this would be remove react-native from peerDependencies to devDependencies

@prtkkmrsngh  Thanks for the suggestion. I have fixed this.


Enable Gantt view public link

Please, do not create duplicate issues
Actual behaviour
Public link works in:
BoardViewController and TaskViewController
Expected behaviour
Enable public link for TaskGanttController (and TaskListController)
Configuration
Plugin version: 1.0.2
Application version: 1.1.1
PHP version: 7.0.22
OS version: Linux 3.2.0-4-amd64
Database driver: sqlite
Database version: 3.14.2
Browser: Browser: Mozilla/5.0

This feature would be really helpful ! :)