Improve performance by parsing function contents lazily

This is a somewhat major rework of when and how functions are parsed and what representation they're stored in.
On my benchmarks, it improves overall find_frames performance by 1.8x to 3x, or, in absolute numbers, from 25-40 seconds to 10-20 seconds. The benchmarks in this crate are improved as well, with two caveats (see further down).
These changes are motivated by these insights:

On current master, there is already one level of laziness: The functions in a unit are only parsed once we encounter an address in that unit. However, in practice, the addresses that are looked up only fall into a small subset of those functions, so we're parsing the contents of many functions unnecessarily. Moreover, this existing level of laziness is not effective in the split debug info scenario, where there's only one unit per object file to begin with.
Parsing names of functions is slow. It takes up memory bandwidth, and, when the data is coming from an mmap'd file, causes more parts of the file to mapped into memory. Parsing function names should be avoided unless the name is actually needed.
The vast majority of DW_TAG_subprogram instances don't cover an address range. They're functions from header files, which are repeated as DW_TAG_inlined_subroutine for the address ranges they actually end up covering in the binary. We were parsing their names unnecessarily.

This PR does the following:

There are now two levels of function parsing laziness.

The first level creates only the function ranges for "outer functions" (DW_TAG_subprogram) and writes down each outer function's dw_die_offset. By necessity, when parsing the list, we need to iterate over all abbrevs and attributes of the unit, including the ones for any nested tags for inlined functions, but we try to keep the work we do for them to a minimum.
The second level is done once we look up an address that falls into an outer function's address range. At that point, we parse the entries for that outer function again, starting at the recorded dw_die_offset. We parse the function name and all functions that are inlined into it, with their names and other data.

This two-level laziness automatically makes it so that we don't parse the names of DW_TAG_subprogram instances that don't have address ranges.
Reduced allocations for inline functions: For nested inline functions, we no longer have a Vec of child inlines on each inline function. Instead, we now have one Vec with all descendant inline_address_ranges on the OuterFunction.
We do a binary search when looking up the inline functions for an address within an outer function.

cargo bench results:
% cargo benchcmp bench-master.txt bench-better.txt
 name                                        bench-master.txt ns/iter  bench-better.txt ns/iter  diff ns/iter   diff %  speedup
 context_new_and_query_location_rc           4,726,113                 3,377,543                   -1,348,570  -28.53%   x 1.40
 context_new_and_query_location_slice        2,003,584                 1,542,540                     -461,044  -23.01%   x 1.30
 context_new_and_query_with_functions_rc     4,668,415                 4,031,275                     -637,140  -13.65%   x 1.16
 context_new_and_query_with_functions_slice  1,953,032                 1,570,379                     -382,653  -19.59%   x 1.24
 context_new_parse_functions_rc              69,957,646                26,982,504                 -42,975,142  -61.43%   x 2.59
 context_new_parse_functions_slice           50,998,880                20,554,205                 -30,444,675  -59.70%   x 2.48
 context_new_parse_lines_rc                  17,311,785                17,292,187                     -19,598   -0.11%   x 1.00
 context_new_parse_lines_slice               11,412,235                11,999,761                     587,526    5.15%   x 0.95
 context_new_rc                              3,192,877                 3,245,348                       52,471    1.64%   x 0.98
 context_new_slice                           1,052,836                 1,054,370                        1,534    0.15%   x 1.00
 context_query_location_rc                   1,126,921                 1,130,779                        3,858    0.34%   x 1.00
 context_query_location_slice                1,171,425                 1,142,100                      -29,325   -2.50%   x 1.03
 context_query_with_functions_rc             3,856,426                 3,803,931                      -52,495   -1.36%   x 1.01
 context_query_with_functions_slice          3,610,648                 3,452,011                     -158,637   -4.39%   x 1.05

The two important improvements are the ones in context_new_and_query_with_functions:
 context_new_and_query_with_functions_rc      -13.65%   x 1.16
 context_new_and_query_with_functions_slice   -19.59%   x 1.24

Caveats:

The parse_functions tests now have a different meaning, so their improvement is artificial. parse_functions longer parses all functions, it now only parses the list of outer functions.
There are some tests that show worse numbers now, especially context_new_parse_lines_slice. This is confusing to me because I don't think I actually changed what happens in them. I think these numbers are a bit flaky and are affected by the file system cache to a large degree. I don't think I actually made anything slower.

I apologize for the size of this PR. The commits aren't the cleanest; some of them don't compile and I sometimes go back and forth between different solutions. The first commit is reverted in the last commit.
If you'd like, I can rewrite the PR as a set with more meaningfully grouped commits. Please let me know if that's necessary. Also, please feel free to make any edits you like.

After these changes, profiling my profiler-get-symbols benchmarks gives me the following breakdowns:
Breakdown of XUL.dSYM benchmark:

35% parsing outer functions list of a unit
30% gimli parsing a unit into list of abbreviations + attributes
10% deallocating context + dwarf data
10% copying data out of addr2lines frame iterator
7.5% parsing an outer function with its inlines
5% parsing lines (excluding render_file)
1.5% render_file

Breakdown for local macOS build with split debug info:

80% parsing outer functions list of a unit
9% parsing an outer function with its inlines
4.5% parsing lines (excluding render_file)
1.5% render_file
1.4% gimli parsing a unit's list of abbrevs + attributes
0.3% dealloc

In the XUL.dSYM case, the biggest potential for improvement would be to have gimli stop allocating its own representation of abbrev + attribute lists, or build them more lazily.
In the split debug info case, there's not much room for improvement, unless there is a way to get the dw_die_offsets of all the subprogram tags without having to iterate over (and parse) all the abbrevs + attributes of any nested DIEs.

These changes sound great! Thanks for doing this.

If you'd like, I can rewrite the PR as a set with more meaningfully grouped commits. Please let me know if that's necessary.

It'd be great if you can do that. It will make it easier for me to review.

The parse_functions tests now have a different meaning, so their improvement is artificial. parse_functions longer parses all functions, it now only parses the list of outer functions.

It'd be good if we had a benchmark for the inner function parsing too, but I can add that later if you want. We won't be able to compare it against the performance in master.

There are some tests that show worse numbers now, especially context_new_parse_lines_slice. This is confusing to me because I don't think I actually changed what happens in them. I think these numbers are a bit flaky and are affected by the file system cache to a large degree. I don't think I actually made anything slower.

Yeah there's a lot of noise in the tests. Running them multiple times might help get stabler numbers.

Breakdown of XUL.dSYM benchmark

This is the one you gave in #171?

Breakdown for benchmark that looks up addresses of a local Firefox macOS build with split debug info:

Do you have any insight into why these are so different? My intuition was that they would be the same, since the dSYM is just all the objects bundled together? Is there any way I can do a similar performance measurement without needing a local Firefox macOS build?

In the XUL.dSYM case, the biggest potential for improvement would be to have gimli stop allocating its own representation of abbrev + attribute lists, or build them more lazily.

I doubt that building them lazily will help unless using DW_AT_sibling lets you skip a significant number of them.

My intuition was that they would be the same, since the dSYM is just all the objects bundled together?

A dSYM only contains a single object file that contains the merged debuginfo of all linked object files together. Also a dSYM doesn't use any debuginfo in the linked executable/library to match functions. Instead it contains the exact offset of every function. DWARF split debuginfo however uses debuginfo in the linked executable/library to match function locations and debuginfo in the split file.

A dSYM only contains a single object file that contains the merged debuginfo of all linked object files together. Also a dSYM doesn't use any debuginfo in the linked executable/library to match functions. Instead it contains the exact offset of every function.

I know the dSYM is a single Mach-O file containing only the debuginfo, but why does that affect performance? We're still only parsing the debuginfo sections in both cases. Isn't the merged debuginfo effectively just the concatenated and relocated .debug_info/.debug_line/etc sections from the object files? If so, why does it make a difference if the debuginfo is in many object files versus contained in a single dSYM?

DWARF split debuginfo however uses debuginfo in the linked executable/library to match function locations and debuginfo in the split file.

What's the relevancy of that? I don't think either of those performance breakdowns were for DWARF split debuginfo?

What's the relevancy of that? I don't think either of those performance breakdowns were for DWARF split debuginfo?

The first breakdown was for dSYM, the second for DWARF split debuginfo I think.

Ah I see. I'm pretty sure macOS doesn't use DWARF split debuginfo, but I could be wrong. I think split debuginfo in the context of that breakdown means using the object files prior to running dsymutil (they are located via the symbol table).

Sorry, yes, when I said "split debuginfo" I meant "debuginfo in object files referenced via OSO stab symbols in the XUL library". What's the right name for that?

If so, why does it make a difference if the debuginfo is in many object files versus contained in a single dSYM?

My guess is that we're just not parsing many of those object files, because they don't contain any of the addresses that we're looking up. So in the dSYM case, when we parse the list of units, we need to iterate over all units in the entire dSYM. But in the object file case, we're only encountering the units in the object files that we know are relevant. So in effect, the OSO symbols act as an index for the relevant dwarf units.

These changes sound great! Thanks for doing this.

If you'd like, I can rewrite the PR as a set with more meaningfully grouped commits. Please let me know if that's necessary.

It'd be great if you can do that. It will make it easier for me to review.

Sure, I'll try to find some time for that. Shouldn't be too much work now that I know what I want the outcome to be.

The parse_functions tests now have a different meaning, so their improvement is artificial. parse_functions longer parses all functions, it now only parses the list of outer functions.

It'd be good if we had a benchmark for the inner function parsing too, but I can add that later if you want. We won't be able to compare it against the performance in master.

Agreed.

Breakdown of XUL.dSYM benchmark

This is the one you gave in #171?

That's right.

Breakdown for benchmark that looks up addresses of a local Firefox macOS build with split debug info:

Do you have any insight into why these are so different? My intuition was that they would be the same, since the dSYM is just all the objects bundled together?

(answered in previous comment)

Is there any way I can do a similar performance measurement without needing a local Firefox macOS build?

I'm working on that. I've uploaded an archive with all the necessary object files to Google Cloud storage now, so I should be able to add a benchmark that uses it.

In the XUL.dSYM case, the biggest potential for improvement would be to have gimli stop allocating its own representation of abbrev + attribute lists, or build them more lazily.

I doubt that building them lazily will help unless using DW_AT_sibling lets you skip a significant number of them.

A big part of the profile is in the allocator. I haven't looked into it too deeply but it looks like gimli is building Vecs with lots of information during this first parse. Sure, we may have to iterate over everything once at the beginning, but maybe it's possible to just create an index with some offsets during the first parse. And then a second parse can be done lazily once the relevant unit is needed. Analogous to what I'm doing with functions in this PR.

[...]  but maybe it's possible to just create an index with some offsets during the first parse.

Oh, we also need the ranges for each unit. So I suppose the real challenge is: When building the initial list of units, get the pc range or range list of a unit without allocating any vectors for that unit. Or something like that.

Oh, we also need the ranges for each unit. So I suppose the real challenge is: When building the initial list of units, get the pc range or range list of a unit without allocating any vectors for that unit. Or something like that.

Ah yep, that's something we could optimize. At that stage we only need the abbreviation for the compilation unit DIE.  This is https://github.com/gimli-rs/gimli/issues/251, but the trait solution I describe there wouldn't be needed now that we're using lower level DIE parsing here.

Another thing that we should do is .debug_aranges support, which would achieve a similar goal.

As a suggestion, I could do the rebase myself and get you to review any changes I make?

I think I'll have time to work on it this coming weekend.

It seems you have little time to work on this, and I do have spare time currently. I've started doing the rebase, and I think it will be a worthwhile exercise, but let me know if that will be a problem.
One thing I would like to do is to see if the RefCell and Rc can be avoided (using LazyCell instead like the rest of the code). Conceptually I think this should be possible.
Also I see that you temporarily had a LocationOnly state for FrameIter. This actually seems like something we should need, and it may be a bug that we're not handling this correctly currently. That is, it should be possible to handle debuginfo that has line number information but not function inlining information.

It seems you have little time to work on this, and I do have spare time currently. I've started doing the rebase, and I think it will be a worthwhile exercise, but let me know if that will be a problem.

Sorry, yes, that sounds great.

One thing I would like to do is to see if the RefCell and Rc can be avoided (using LazyCell instead like the rest of the code). Conceptually I think this should be possible.

Definitely worth trying! This was actually my first time writing code that uses RefCell and I wasn't entirely sure if it was a good choice. But conceptually it does make sense to me to have shared ownership of the function data between the iterator and the cache.

Also I see that you temporarily had a LocationOnly state for FrameIter. This actually seems like something we should need, and it may be a bug that we're not handling this correctly currently. That is, it should be possible to handle debuginfo that has line number information but not function inlining information.

Sounds good. I think I added it when I thought "oh no I broke this case", only to later realize that the original code didn't handle it either (I think), so I removed it again because I didn't want to introduce behavioral differences in an optimization patch.

Sounds good. I think I added it when I thought "oh no I broke this case", only to later realize that the original code didn't handle it either (I think), so I removed it again because I didn't want to introduce behavioral differences in an optimization patch.

I didn't pursue this further. I agree that would be better for a new patch.


Avoid using transformed parameters in GLM

Closes #16
This moves the code for mu and mu_unlinked into model and generate quantities. There is no reason we need to save the output of this, so this should be equivalent for our use cases.

Makes sense to me and I completely agree with the reasoning in the linked issue. Thank you!


Upgrade to Java7

Addresses #373.
Is this the right way to perform the upgrade w.r.t Gradle?  /cc @rspieldenner @quidryan

Hystrix-pull-requests #249 FAILURE
Looks like there's a problem with this pull request

NetflixOSS » Hystrix » Hystrix-pull-requests #10 SUCCESS
This pull request looks good

Merging this now, as it passes all tests.  If there's further work around Gradle setup, please open a new issue

you don't need the sourceCompatibility and targetCompatibility = 1.7 as those are the defaults but it won't hurt anything

Great, thanks for the confirmation @rspieldenner


Why need such complex code instead of one line?

myImage.Source = new BitmapImage(new Uri("smiley.png"));

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 71fdbaa6-085d-0162-2015-3661c60dd93b
Version Independent ID: 399ebd64-229f-5684-4803-29de42008e80
Content: How to: Encode and Decode a PNG Image
Content Source: docs/framework/wpf/graphics-multimedia/how-to-encode-and-decode-a-png-image.md
Product: dotnet-framework
Technology: dotnet-wpf
GitHub Login: @dotnet-bot
Microsoft Alias: dotnetcontent

Thank you for the question. The point of the article is not to display the image, although it does show that, the point is to learn how to use the types related to decoding and encoding an image in PNG format.


Better-looking default skin

IMO the hitobjects should be in a similar style to osu!lazer's skin.
The regular minion could look something like this (rough sketch done in Krita):

Yeah the minions we have are just placeholders as we care for the mechanics of the game more than the style currently.
@swoolcock The design looks nice to me, but would like to take your opinion into account as well.

Would need to change the player sprite then too, otherwise it would look weird.

@frenzibyte @recapitalverb any suggestions on the player sprite then?

I'll try sketching out some ideas I have soon.

My artistic skills apparently disappointed me, so I opted for something simpler; I think we could go for the stickman seen in the Rush! logo in README.md:

To implement the above in the game, it's probably okay to store animations in monochrome and then use that as a mask for an instance of  osu.Game.Graphics.Backgrounds.Triangles, though I'm not very familiar with osu!framework.

@recapitalverb do you have the file for that triangle minion you could share?

It was a mere sketch so I didn't have the project file, sorry. Though you could just use osu.Game.Graphics.Backgrounds.Triangles and a clipping container with outline and glow, as mentioned above. Building from components already in osu!lazer and !framework are better here anyways.

Yeah, I'm not sure how simple it is to make rounded corners on a triangle though.  I'll fiddle with it a bit and see what I can do.


Update H2 database

This fixes this security issue:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42392.
More info here:
https://github.com/h2database/h2database/security/advisories/GHSA-h376-j262-vhq6

This should have a Jira case associated with it (and prefixed in the commit and PR title). Also is there a reason why only the Registry's version of H2 is being updated rather than all the products (nifi-database-utils, minifi, etc.)? The last time I looked at updating the H2 version, there were issues with INTEGER fields without precision, as well as generic ARRAY types (with no element type specified). Not sure if those issues affect Registry but I'd rather update H2 across the board when there's a version that doesn't have breaking changes (such as a 1.4.x version that includes the CVE fix).

The unit tests are failing because Flyway doesn’t like the H2 version, or the SQL is no longer compatible (the latter of which I ran into with NiFi when trying to upgrade H2). Did you go through the PR template steps?

Closing this pull request since a simple version upgrade breaks compatibility across several components as shown in the automated builds. A new Jira issue, along with additional changes to impacted components, will be necessary to upgrade to the latest version of H2.


Cleaned up UInt32 parsing APIs

Trying to settle on the shape of the parsing APIs. This is the first stab implementing uint parsing.

@sokket could you take a look?

Now when there is ReadOnlySpan shouldn't it be used in public API?  Where appropriate.

Good point. I will update to ROSpan

LGTM

@dotnet-bot test this please

@dotnet-bot test this please


fixes after scalapb upgrade

Description
Build fixes for scala 2.12 and strict_deps enabled

Motivation
Build failures in my current project after last PR (#1026)

I think it's a problem:
CI fails with dependency_tracking_method = 'default' and without explicit protobuf dependency;
my local build fails with dependency_tracking_method = 'high-level' and with explicit dependency from master.

Sorry for the delay. Corona times are wrecking a havoc in my availability.
I think that the best thing to do is to revert your change and then push a new commit that only adds the failure that you're seeing locally to CI. Then we can try and devise a solution.
I'll try to do it now.

cc @Jamie5 I have a hunch that somewhere in the flag changes we lost some of the e2e tests we used to have

btw, worst case I think we'll be able to add the problematic dep to the ignore targets of scalapb_generator_lib.
Let's hope we wont' reach this.

Also right now dependency_tracking_method='default' means dependency_tracking_method='high-level'.
It would really help to get a repro or even just your full setup.
What toolchain are you using (one of the default ones or a custom ones), what scala_repositories and which target fails (I'm assuming it's the same one that's failing right now but would love to make sure).

btw, it would also help to see the exact failure

Closing in favor of #1030 which has a workaround for both issues.
Feel free to reopen if you have strong objections.

btw, this isn't time critical or anything, right?
Asking since I'd like to give Jamie time to review #1030 and we don't have anything newer than the scalapb upgrade so I'm guessing you can wait but would still like to verify

I can wait for #1030 to be merged


fix: Id of an entity is used for entityId instead of processInstanceId

[ ] CloudSequenceFlowTakenImpl
Not sure if here it is correct
Related tests:
/activiti-cloud-runtime-bundle-service/activiti-cloud-services-runtime-bundle/activiti-cloud-services-connectors/src/test/java/org/activiti/services/connectors/channel/ServiceTaskIntegrationResultEventHandlerTest.java

[ ] CloudIntegrationResultReceivedImpl
Related tests:
/activiti-cloud-runtime-bundle-service/activiti-cloud-services-runtime-bundle/activiti-cloud-services-connectors/src/test/java/org/activiti/services/connectors/channel/ServiceTaskIntegrationResultEventHandlerTest.java

[ ] CloudBPMNActivityEventImpl
Dependent:
CloudBPMNActivityCancelledEventImpl
CloudBPMNActivityCompletedEventImpl
CloudBPMNActivityStartedEventImpl
Related tests:
/activiti-cloud-audit-service/activiti-cloud-starter-audit/src/test/java/org/activiti/cloud/starter/audit/tests/it/AuditServiceIT.java

[ ] CloudIntegrationRequestedImpl
Related tests:
/activiti-cloud-runtime-bundle-service/activiti-cloud-services-runtime-bundle/activiti-cloud-services-connectors/src/test/java/org/activiti/services/connectors/IntegrationRequestSenderTest.java


AnimeList entry deletion returns 500

I may be executing this request incorrectly, but it appears that when this request is sent with the DELETE verb, it returns a 500. Is that the correct placement of the AnimeID? When I attempt to call this on an ID that is located on my "completed" list (verified by executing the same command with a GET), it returns XML in the shape of a server exception message.
The request in question:
http://anilist.co/api/animelist/1?access_token=$ACCESS_TOKEN

Make sure you are including the appropriate Authorization header in your request.

Now to access the resource server on the resource owner’s behalf, simply include the following header with all your requests

Authorization: Bearer access_token

I'm fairly certain the request is well formed. I use the same authentication Megan's
method for everything else, and that all work fine.
Note that I'm not getting an unauthorized response. I'm getting a 500 in the form of a page redirect (the response is an html document, not JSON)

Yeah, you're getting the "Woops, looks like something went wrong" error, right?

I only get that if I don't include the header or the item wasn't on my list in the first place.

That is absolutely bizarre. I was absolutely certain that it was well-formed, but lo-and-behold, it was missing the capital B. Oh well, thanks Kaz.


execution engine: fix for-else clause for Threshold condition parsing

I'm not sure how this was missed but on each ci test run, the related test test_threshold is not run because the graph-scheduler version is outdated.

@kmantel Was it the intention to open the PR to main? Also, tests are not passing.

@kmantel Was it the intention to open the PR to main? Also, tests are not passing.

Yeah, this is a bug that should probably get in sooner. The tests are not passing due to an unrelated problem with pytorch 1.12 on the CI, which will affect other runs as well. I can't find exactly when this was changed but this references the problem: https://github.com/microsoft/onnxruntime/issues/11901


Change the extension of html to php

Is there any way to change the front-end of the WebRTC video conference HTML page into .php extension instead of .html extension?
As i would like to connect it to the database. Thanks

Hello @yanzai3292
Yes and no :)
The codelab is designed to leverage node.js. By default, Node.js only supports Javascript as the backend language. This means, no php files. There are, however, ways to make PHP work on Node.js (example).
Now, WebRTC itself does not require Node.js. The reason for which we used Node.js is for signaling and allowing peers to connect to each other (article). However, if you can replicate this signaling infrastructure in a different server, it would work and you would be allowed to do it. I have not tried, but I found this repository where they try to accomplish what you want, I believe.
I hope this helps.
/cc @samdutton

Hi @yanzai3292 — as @nitobuendia says, this codelab is oriented to Node, but you can certainly do WebRTC from a PHP page.

Hi @yanzai3292 — as @nitobuendia says, this codelab is oriented to Node, but you can certainly do WebRTC from a PHP page.


fix: handle reference margins and 1px error

This will likely need to be refactored into something better but this is the general idea to fix the issue.

It's for things like that that the v1 code base is so complicated 😭
We need to take some time and figure out a clean solution to the problem.

Well I think the issue is reference element's margins shouldn't be included in its rect (only popper should).
And we need to round the values to prevent 1px errors for variation placements.

I manually merged the fix for the reference margins. Could we keep this PR just for the rounding issue?


Log URLs listed in Qualified URLs don't work

https://ct.googleapis.com/pilot
https://ct.googleapis.com/aviator
....
these URLs don't exist.

They do. However, they are the base URLs for the RFC 6962 API. They are not URLs you can perform a bare GET over / browse.


IEA annotations should be broken out by reference in stats

IEA annotations should be broken out by reference (this is the method used), and converted to a provider, e.g. GO_REF:0000019 is Ensembl Compara.
Originally requested by Paul T

This essentially requires that the AmiGO stats page either 1) becomes dependent on another repo (go-site) or 2) the labels and the like are loaded into GOlr.
The second is preferable, but will take more jiggering.


TSR: Invalid elimination for an Object destructuring of an import case

Part of https://github.com/TanStack/router/pull/1837, moving TanStack Router's plugins to use babel-dead-code-elimination.

For reference, this is our current dead code elimination process that currently works!
For reference, this is how we are consuming the babel-deal-code-elimination package on our end

This is the test case that is failing.
Input:
import thing from 'thing'

export function test() {
  const {
    foo: {
      bar: { destructured },
    },
  } = thing

  return destructured
}

Current output:
export function test() {
  return destructured;
}

Expected output:
import thing from 'thing';
export function test() {
  const {
    foo: {
      bar: {
        destructured
      }
    }
  } = thing;
  return destructured;
}

This is the original test file should it prove useful.

Fixed by #12


[website] Update team info

Add new committers' info on the Pulsar website.

@315157973 @yjshen welcome onboard~
Please help check your info, thank you.


[0.11] Empty graphiql.min.js

Hi,
Looks like the most recent build is missing graphiql.min.js file in the distribution. That is<EMAIL_ADDRESS>is empty but<EMAIL_ADDRESS>is fine. We reference it from the CDN as <script src="//unpkg.com/graphiql/graphiql.min.js"></script> which now redirect to 0.11, and thus breaks some of the pages.
Is it possible to still ship the .min file?

Thanks for reporting! We should be able to cut a corrected release for this shortly.

I've fixed the build, so will cut a new release presently.

v0.11.1 is now out on NPM and you can see the min build at<EMAIL_ADDRESS>

Seems to not have a min file in unpkq in v0.11.2


Package the front in the python package as a clip-retrieval front

keep or not the npm packaging, but it would be quite convenient if it was also bundled in the python package
it would make it easy to make clip-retrieval end2end  do stuff then boot a frontend with no additional user manipulation
related #31

and maybe default to localhost as a back too

done


Some HDF5 compatibility tests randomly fail

In GitLab by @jbigot on Nov 15, 2020, 20:32

mentioned in commit b0181cdd0b24f76c69436c646715e175b7f6c0ee

mentioned in commit f8ec12a14ceb3ccb70046e3483993a6b65ee6e4c

mentioned in commit 32ab5202d45d68d53e11d153b10da2ff5195e298

mentioned in commit e28cfc5ca1736a06b2bba98c5ed70201a9a167e1

mentioned in commit 7b95e166d76af964be8ba11997d8d1af3f3350f7


fix humanoid fire visuals missing

About the PR

humanoids used to use the more intense fire sprite after reaching fire level 4. It went gone after some rework it seems. I add it back.
Why / Balance

fix
Technical details

The parent to all humanoid species now gets the texture for alternate state. For definition of the FireVisuals component in question, see MobFlammable.
Media
✅ Humanoids use the sprite
✅ Other mobs do not use the sprite

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

n
Changelog

god bless america


Publish pre-releases from master builds / Trigger dependant client libraries workflows to raise PR's

The current ecosystem is large and we are currently growing the number of client libraries consuming the pact ffi library.
In some cases, the test suites of the consuming the pact ffi library can throw up errors, not evident in this repo's test suite.
I propose a couple of measures that could improve the feedback loop.

On release of Pact FFI  / Verifier CLI / Mock Server CLI, raise a PR against the consuming libraries repositories, updating to consume the new library

Ecosystem Map
Process in Ruby

Trigger
PR Raising Workflow  & Script

In order to get quicker feedback loops, prior to an official release, we could pre-release against pushes to master, which could trigger the respective client libraries workflows. Maybe this would be better named next or something

Thoughts on a postcard. I would really like to do point 1, that brings us to line with the old pact ruby core.
Point 2 should be considered, maybe not the implementation but that forward feedback, I feel would be valuable for myself, and if anything was uncovered in the respective client libraries that could be found earlier in this repo, we could add additional test cases.

I suppose that just raises another question we should consider - what do we need to do to get it to a 1.x.x?
Perhaps that can be the topic of an upcoming maintainer meeting?

Thanks Adam, that is really useful feedback as always.

It's very hard to know release-to-release exactly what's changed, and given FFI is still at 0.x it just changes really rapidly. Functions get deprecated or 'v2' versions added, even in some cases the entire interaction model has changed between versions. I wouldn't want to inherit all of this churn frequently.

The release notes often say "upgraded pact-models to 0.1.2" or whatever, which means you have to follow the change logs into all the other libraries as well to work out what's included in a new FFI version.

Even then sometimes you need to actually go and read all the code because the commit message doesn't tell you what changed or why - I've literally just finished checking the code for a commit with the message "stupid Windows", which isn't helpful to me at all.

Yep, I can empathise with this. I do think there is some improvements that we make so provide a summary of the changes, outside of the commit logs. I do find it a bit of a chore to traverse the release notes, with commit refs and tracing them down to individual crates, and working out the changes from commit messages, without reading the code.

I wouldn't want to inherit all of this churn frequently

Apologies, my intention wasn't really to force the maintainers of the client libraries to inherit this churn, but to provide a quicker feedback loop between releases of the pact-ref project, and the impact on client libraries (and conversely maintainers)
Would a release tracking issue, that has a summary of the release contents, impacts/changes for maintainers be useful, alongside a place to discuss any issues, and provide links to PR's in consuming repos. I'm wondering if seeing a PR of the new release consumed in another client library, help you in some way as a maintainer?

It's a weird chicken-and-egg thing, because effectively you want to test pre-release versions of the FFI against all the client libraries to spot any issues early, but the client libraries don't want to accept pre-release versions of the FFI, only 'full' releases. But waiting until they're properly released is just asking for finding breaking changes too late 😁
A PR turning up really depends on what the contents are. If it's a bump from 1.2.3 to 1.2.4 then no big deal. We're only expecting bug fixes, there shouldn't be any functional change at all, and certainly no breaking changes. Those should pretty much fly through.
Any minor or major change would be more invoved though, and needs to come with some kind of summary of what's changed and what impact that may have, so that's when more detailed release notes are needed.
I've been going through the FFI this afternoon updating PactNet from 0.4.5 to 0.4.16, for example, and lots of things have changed. I've replaced something that was now deprecated, but other bits I'm confused as to how they're even supposed to be used, and the docs/examples often still refer to the deprecated bits so I've basically just ignored the new confusing bits for now.
That's a related problem to this one though, because the 'best practice' usage guide of the FFI needs to be really obvious, and changes to that called out in a clear way each time they happen. Once we get to that point then raising PRs automatically with those changes included becomes a possibility.
As for the actual question of what those PRs contain, that's a tricky one. When a method is deprecated or just no longer recommended (I'm thinking the v2 versions of methods, or the multiple different interaction models that now exist), would we expect the PR to contain the C# (for example) changes required to support them? If not, and the C# continues using the old deprecated versions of things, I'm unclear what the value in raising the PRs is, because the new stuff isn't being tested anyway.
So, automated PRs for bugfix releases (once we're at 1.x and beyond) makes sense to me. Automated PRs for major/minor releases I think is much more tricky, and in general I think that entire process probably needs some more thought for maintainers more like myself who don't work for Pactflow/Smartbear/etc and are instead volunteers. The FFI churn is currently high and it's very easy to lose track of what's changing and what you need to do in response currently.

As for the actual question of what those PRs contain, that's a tricky one. When a method is deprecated or just no longer recommended (I'm thinking the v2 versions of methods, or the multiple different interaction models that now exist), would we expect the PR to contain the C# (for example) changes required to support them? If not, and the C# continues using the old deprecated versions of things, I'm unclear what the value in raising the PRs is, because the new stuff isn't being tested anyway.

Deprecations and new features/methods will always happen with libraries, I don't think that should be the reason we wouldn't auto-open a PR even in such cases where the only changes will be these sorts of things (i.e. no additional value for the client project). In most cases, the release will include changes that fixes bugs or adds improvements to the existing features.
I think the key point would be ensuring the key new features / changes are linked along with the PR. That way, the commit could contain breadcrumbs to all of the changes (instead of the more opaque "fix: update FFI core to version x.y.z" type thing).

Yeah that's why it needs to be at 1.x first though. You don't need to bump the version just to deprecate something, but you do need to raise a major version change in order to actually remove them.
If a PR is raised and contains both bugfixes and breaking changes (which 0.x releases always allow) then the PR process in general has much more friction. If I want the bugfixes I have to take the breaking changes at the same time, and they may be non-trivial. That delays getting fixes out for client libraries, for example.
If it was at 1.x then we can produce both a 1.0.1 with just the fixes to allow client libraries to upgrade quickly, and a 2.0.0 to introduce the breaking changes separately in a more considered way. If any new fixes are needed we can produce 1.0.2, 1.0.3 etc. whilst we're waiting for client libraries to migrate to 2.x. That's much less friction than the all-or-nothing 0.x approach.

As discussed in the maintainer meet yesterday, here are my thoughts on this issue:
FFI is already a de-facto 1.x.x
Multiple languages now rely on it (JS, Go, .NET and soon Python and PHP), so any breaking changes between patch versions are felt regardless of the actual naming scheme.
Josh raised that there are some inconsistencies in FFI functions

Return arguments may be void, bool or int.
Mixed use of pointer types and handles
Mixed use of logs vs fetching via string helpers

(@JP-Ellis I probably have gotten some of these points wrong, please correct me here / edit this post as needed).
We may want to address these - and any others we believe should be tackled now - in a single hit, and then pull the trigger on a 1.x.x. This will give a stable upgrade path a go.
Opaque Release Notes
As Adam raised above, one challenge of simply bumping the FFI version in a client language is that it's not clear to end users what a given version of the FFI brings to the changelog of that client. It makes it harder for them to see what's new, trace the source of bugs etc.
If the PR could contain the upstream changes (that might be in turn, collected from other pact-reference changes) that would improve this situation.
I agree with this point.
Equally, however, I don't think that should stop us from automating the PRs assuming we had a stable release line and pushed backwards compatible changes. The Ruby updates didn't contain this, and were universally welcomed by maintainers. In some cases, regression suites in the client languages turned up a problem, which meant we could quickly revert or patch breakages. If nothing else, this is a great reason to institute this change.
New feature awareness for the maintainer
The corollary to the above point is to improve visibility for the maintainer. Perhaps sharing a diff of the headers file could be a way to scan for new major features that are now available to the maintainer that could spark them to create a feature request knowing the capability now exists.

You've summarised my points quite well!
Specifically regarding the inconsistency in the interfaces, while I think it would be nice to have to a 1.0.0 release, this would also be a significant breaking change for the downstream libraries. I think such a change would also require community input to make sure we have a mutually agreeable standard.
So even if we did mark what we have now as 1.0.0, the above can still be achieved at a later time.


Qwen2-0.5B-Instruct lora微调报错

Reminder

[X] I have read the README and searched the existing issues.

System Info

llamafactory version: 0.8.3.dev0
Platform: Linux-3.10.0-1160.114.2.el7.x86_64-x86_64-with-glibc2.35
Python version: 3.10.6
PyTorch version: 2.1.0a0+b5021ba (GPU)
Transformers version: 4.42.4
Datasets version: 2.20.0
Accelerate version: 0.32.1
PEFT version: 0.11.1
TRL version: 0.9.6
GPU type: NVIDIA GeForce RTX 4090

Reproduction
CUDA_VISIABLE_DEVICES=2 llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml

config:

### model
model_name_or_path: model_hub/qwen/Qwen2-0.5B-Instruct

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all

### dataset
dataset: zhidian_data
template: llama3
cutoff_len: 1024
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 8

### output
output_dir: saves/qwen/Qwen2-0.5B-Instruct/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500

problem:
07/15/2024 06:49:29 - INFO - llamafactory.cli - Initializing distributed tasks at: <IP_ADDRESS>:25148
[2024-07-15 06:49:32,277] torch.distributed.run: [WARNING]
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
07/15/2024 06:49:43 - WARNING - llamafactory.hparams.parser - `ddp_find_unused_parameters` needs to be set as False for LoRA in DDP training.
07/15/2024 06:49:43 - INFO - llamafactory.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
[INFO|tokenization_utils_base.py:2159] 2024-07-15 06:49:43,005 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2159] 2024-07-15 06:49:43,006 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2159] 2024-07-15 06:49:43,006 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2159] 2024-07-15 06:49:43,006 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2159] 2024-07-15 06:49:43,006 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2159] 2024-07-15 06:49:43,006 >> loading file tokenizer_config.json
[WARNING|logging.py:313] 2024-07-15 06:49:43,341 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/15/2024 06:49:43 - INFO - llamafactory.data.template - Replace eos token: <|eot_id|>
07/15/2024 06:49:43 - WARNING - llamafactory.data.template - New tokens have been added, make sure `resize_vocab` is True.
07/15/2024 06:49:44 - WARNING - llamafactory.hparams.parser - `ddp_find_unused_parameters` needs to be set as False for LoRA in DDP training.
07/15/2024 06:49:44 - INFO - llamafactory.hparams.parser - Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
07/15/2024 06:49:44 - WARNING - llamafactory.hparams.parser - `ddp_find_unused_parameters` needs to be set as False for LoRA in DDP training.
07/15/2024 06:49:44 - INFO - llamafactory.hparams.parser - Process rank: 2, device: cuda:2, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/15/2024 06:49:44 - INFO - llamafactory.data.template - Replace eos token: <|eot_id|>
07/15/2024 06:49:44 - WARNING - llamafactory.data.template - New tokens have been added, make sure `resize_vocab` is True.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/15/2024 06:49:44 - INFO - llamafactory.data.template - Replace eos token: <|eot_id|>
07/15/2024 06:49:44 - WARNING - llamafactory.data.template - New tokens have been added, make sure `resize_vocab` is True.
[9bc5436f4dd8:2204 :0:2311] Caught signal 7 (Bus error: nonexistent physical address)
[9bc5436f4dd8:2203 :0:2307] Caught signal 7 (Bus error: nonexistent physical address)
[9bc5436f4dd8:2205 :0:2312] Caught signal 7 (Bus error: nonexistent physical address)
==== backtrace (tid:   2311) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x000000000008def3 ncclGroupEnd()  ???:0
 2 0x0000000000094f1c ncclGroupEnd()  ???:0
 3 0x000000000006e41c ncclCommAbort()  ???:0
 4 0x000000000005cc27 ncclCommAbort()  ???:0
 5 0x000000000005fe94 ncclCommAbort()  ???:0
 6 0x000000000007912c pncclRedOpDestroy()  ???:0
 7 0x0000000000094b43 pthread_condattr_setpshared()  ???:0
 8 0x0000000000125bb4 clone()  ???:0
=================================
==== backtrace (tid:   2307) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x000000000008def3 ncclGroupEnd()  ???:0
 2 0x0000000000094f1c ncclGroupEnd()  ???:0
 3 0x000000000006e41c ncclCommAbort()  ???:0
 4 0x000000000005cc27 ncclCommAbort()  ???:0
 5 0x000000000005fe94 ncclCommAbort()  ???:0
 6 0x000000000007912c pncclRedOpDestroy()  ???:0
 7 0x0000000000094b43 pthread_condattr_setpshared()  ???:0
 8 0x0000000000125bb4 clone()  ???:0
=================================
==== backtrace (tid:   2312) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x000000000008def3 ncclGroupEnd()  ???:0
 2 0x0000000000095106 ncclGroupEnd()  ???:0
 3 0x000000000006e7d6 ncclCommAbort()  ???:0
 4 0x000000000005cc27 ncclCommAbort()  ???:0
 5 0x000000000005fe94 ncclCommAbort()  ???:0
 6 0x000000000007912c pncclRedOpDestroy()  ???:0
 7 0x0000000000094b43 pthread_condattr_setpshared()  ???:0
 8 0x0000000000125bb4 clone()  ???:0
=================================
[2024-07-15 06:49:52,304] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -7) local_rank: 0 (pid: 2203) of binary: /usr/bin/python
Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.1.0a0+b5021ba', 'console_scripts', 'torchrun')())
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 797, in main
    run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 788, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
====================================================
/app/src/llamafactory/launcher.py FAILED
----------------------------------------------------
Failures:
[1]:
  time      : 2024-07-15_06:49:52
  host      : 9bc5436f4dd8
  rank      : 1 (local_rank: 1)
  exitcode  : -7 (pid: 2204)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2204
[2]:
  time      : 2024-07-15_06:49:52
  host      : 9bc5436f4dd8
  rank      : 2 (local_rank: 2)
  exitcode  : -7 (pid: 2205)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2205
----------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-15_06:49:52
  host      : 9bc5436f4dd8
  rank      : 0 (local_rank: 0)
  exitcode  : -7 (pid: 2203)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2203
====================================================

Expected behavior
No response
Others
No response

发现是我的CUDA_VISIBLE_DEVICES写错了！

您好，请教一下为什么你的路径是这个model_name_or_path: model_hub/qwen/Qwen2-0.5B-Instruct？
我现在用Qwen/Qwen2-0.5B-Instruct报错

model_hub/qwen/Qwen2-0.5B-Instruct是我本地的路径，我把模型提前下好了，你使用模型名字的话大概率会连接huggingface下载模型，但是国内的网和huggingface是不通的，所以报错


[origin/wip_warrior_tx2] bzip2 fails to extract JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2

This issue isn't directly related to the balena-jetson-nano project, but it is encountered during a normal jetson-tx2 build so I wanted to document my workaround and see if there were more elegant options.
It seems the L4T Driver Package (BSP) provided by NVIDIA does not extract correctly with current versions of bzip2. This causes the bitbake build to fail.
$ wget https://developer.download.nvidia.com/embedded/L4T/r32_Release_v1.0/jax-tx2/BSP/JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2

$ sha256sum JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2
e7ecaa32928484398d2492e6e9911262b4663e720b4867404d8fe1928fe483aa  JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2

$ tar xjvf JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2
Linux_for_Tegra/
Linux_for_Tegra/nv_tegra/
Linux_for_Tegra/nv_tegra/LICENSE.weston
Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2

bzip2: Data integrity error when decompressing.
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

$ bzip2 -d JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2

bzip2: Data integrity error when decompressing.
        Input file = JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tbz2, output file = JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tar

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

bzip2: Deleting output file JAX-TX2-Jetson_Linux_R32.1.0_aarch64.tar, if it exists.

$ tar --version
tar (GNU tar) 1.29

$ bzip2 --version
bzip2, a block-sorting file compressor.  Version 1.0.6, 6-Sept-2010.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

My current workaround is as follows:

manually decompress the archive on another PC with just tar & lbzip and not a full bzip2 binary
recompress the archive and check for integrity errors
host the new archive locally and patch tegra-binaries-32.1.0.inc to point to the new file and hash

Has anyone else encountered this? I imagine most people have this file cached so it doesn't get redownloaded on an existing workspace.
If anyone has a more elegant workaround I'd love to hear it. I will likely open a ticket with NVIDIA to see if they can resolve the issue with the uploaded packages.

Hi @klutchell
In another ticket you mentioned issues with signing binaries for tegra186-flash-dry. This means that tegra-binaries - which brings this L4T archive - had been downloaded and unpacked at that point.
But probably you tried out a new build now and the archive is downloaded, but can't be unpacked anymore.
Many people have reported this exact same issue these days on the nvidia forums: https://devtalk.nvidia.com/default/topic/1056381/jetson-agx-xavier/jetpack-4-2-xavier-install-failed/2
and the current workaround presented there is to downgrade bzip2 and libbz2 on the host side. We'll see what the official response is and take the appropriate steps.

Thanks for the link to the NVIDIA forums issue, I was just in the process of searching for that. My workaround of hosting the package locally is working for new builds so I'll continue to do that until NVIDIA fixes the issue on their side.

Also discussed here: https://devtalk.nvidia.com/default/topic/1048691/jetson-tx2/jetpack-4-2-mdash-l4t-r32-1-release-for-jetson-agx-xavier-jetson-tx2-and-jetson-nano/4 but no official reply yet. Worth keeping an eye on both threads.


Scrapping data from twitter

Is your feature request related to a problem? Please describe.
Twitter is used by millions of users to display their take on various issues ranging from micro to macro level. I would like to scrape tweets regarding important topics to analyse the take of people over different issues.
Describe the solution you'd like
I am thinking to create a dataset by scrapping all the information regarding the account, likes, bookmarks, retweets and followers so on using official twitter api.
Link from where I want to scrap: https://twitter.com/home
Please allow me to work on this under GSSOC'23

Sorry, we are not using any APIs in this project.


[cosmetic] Weather module humidity positioning

This PR adds an option to tweak the layout of the weather module. When set, the humidity appears alongside the temperature:

Thank you for the PR.
We can only accept PR's based against our develop branch, we can sadly not enforce this technically.
So please create a new PR againt develop instead of master:

@khassel Sorry! I had based my branch against develop but somehow the PR came through with the wrong target. PR updated (please let me know if you need a fresh PR instead).

Also thanks from me for the PR. However I would propose to change it so that the position of the humidity is controlled via the showHumidity config parameter and not introduce a new one.
For that showHumidty would be changed to a string (with options "below" and "besideTemp" (but those names are not optimal I think)) and a deprecation warning for the boolean value.
See https://github.com/MagicMirrorOrg/MagicMirror/blob/8c0e7db494e86895a83a854cf1071c9057098ce3/modules/default/weather/weather.js#L83 for an ecample.

Also thanks from me for the PR. However I would propose to change it so that the position of the humidity is controlled via the showHumidity config parameter and not introduce a new one.

Thanks, I like that idea. I'll have a think about how to express that.

please let me know if you need a fresh PR instead

no, looks good so far (except one test)

OK, see what you think of this. showHumidity is now a string specifying one of a handful of possible positions.
Supported values are wind, temp, feelslike, below, false  (where "false" means not displayed).
We also recognise true and false (boolean, unquoted) as legacy values and emit a deprecation warning.
(I have force-pushed this branch, I hope that the PR checks recognise that it is now based off of develop. If not I may need to redo with a fresh PR.)

test fails are unrelated to this PR

Rebased back onto develop

Looks good, just a minor thing. Also, would you be able to write a test for this?

I've added a test now. I'm not sure how you feel about embedding a raw non-breaking-space character in the test string though? There might be a better way to do it, or maybe a tweak to the the template would leave things tidier.

I'm not sure how you feel about embedding a raw non-breaking-space character in the test string though?

Replaced the embedded space with its escape string \xa0.

nearly there, good work nonetheless!

Thanks for the feedback! I'm more at home with C++ than I am with js, it probably shows...


FakeVive, EarthVR and OSVR HDK

Have been enjoying steamvr content using an osvr hdk for hmd.
EarthVR can not launch even after using FakeVive due to the hardware check.
Does freepie or something need to be used to trick earth vr into thinking vive controllers are connected too?
The whole point of openvr being in existence is so that users of an array of vr devices can enjoy vr content regardless of what devices that vr content officially supports.
I am shocked Valve have allowed an app to be released that totally breaks openvr support.
With reference too https://www.reddit.com/r/Vive/comments/5dcbgi/how_to_remove_the_google_earth_vr_headset_check/ in step 4 Vive hmd = 84 and Rift = 81so we just need to work out what OSVR HDK =   ?.
How to get OSVR HDK working with EarthVR?

Does your EarthVR directory look like this?

Does freepie or something need to be used to trick earth vr into thinking vive controllers are connected too?

EarthVR doesn't check the controller model. People have been using it with Oculus Touch and Razer Hydra controllers without any problems.

With reference too https://www.reddit.com/r/Vive/comments/5dcbgi/how_to_remove_the_google_earth_vr_headset_check/ in step 4 Vive hmd = 84 and Rift = 81so we just need to work out what OSVR HDK = ?.

That's actually just editing the machine instructions involving the HMD check. The number isn't specific to any hardware, it's just changing the control flow of the program.

The latest release compiled binaries zip was used and my earthvr install directory looks like

Been trying different things but earthvr will not launch

Hmm, it looks like you have FakeVive set up correctly. Have you tried using the original method of hex editing the Earth.exe file?
When you say "earthvr will not launch" - are you seeing a message box that exactly says "Earth VR only supports HTC Vive at this time"? Or is the program just crashing or refusing to start?
@LibreVR, would you happen to know anything about this? I'm not too familiar with OSVR.

I used the original instructions found at https://www.reddit.com/r/Vive/comments/5dcbgi/how_to_remove_the_google_earth_vr_headset_check/ and could not find the described "84" value to edit; which was after trying FakeVive.     A fellow OSVR HDK user posted info which has given a working EarthVR with OSVR HDK 1.4.  http://oculusrift-blog.com/google-earth-vr-was-updated-heres-how-to-remove-the-headset-check-again/9005/
The difference is searching for "0x72520" or "0x727D0"

Issue solved and closed.   :-)

I'm glad that this works for you, but the issue still remains that FakeVive wasn't working. If Google updates the program again then you'll be stuck having to patch the EXE again.
Re-opening. I'll look into some alternate code injection methods, perhaps ddraw.dll isn't getting loaded from your EarthVR directory or something.

@Balderick Actually that was me, not a fellow OSVR HDK user, seems like that is a site that just carbon copies things off reddit.
Have you tried the latest version of FakeVive? I think the OSVR driver doesn't report some properties, which is fixed in the latest version.

After trying latest version FakeVive multiple times get the same outcome.  "Headset not supported"
I have to manualy edit the hex file to get EarthVR to launch.

I've published a new release, FakeVive 1.2, to help diagnose this. It's probably not going to fix anything, but it will provide a log file which we can hopefully use to debug this further.
When you have time, please do the following:

Update to the new release.
Make sure to restore your EXE to the original version. If you don't have a backup, you can have Steam redownload it: right-click Google Earth VR, go to Properties, go to the "Local Files" tab, and click "Verify Integrity of Game Cache".
In Steam, right-click Google Earth VR, go to Properties, click "Set Launch Options", enter -FakeViveDebug, and press OK.
Run Google Earth VR. At some point you should see a console window pop up with debugging info inside it. If you don't, then let me know.
Open Windows Explorer, type %TEMP% into the address bar, and press Enter. You'll be taken to your Temp folder. Look for a file in there named FakeVive.log and attach it to a reply.
If you don't see a FakeVive.log file, try running Earth.exe from outside of Steam and then look again. Make sure to let me know if you did this.

After updating to FakeVive 1.2 no FakeVive.log gets created and -FakeViveDebug option does not show console window.
Tried with steam running and directly from EarthVR install location.
Searched whole system for FakeVive.log but nothing.

OK, so this confirms my suspicion that the ddraw.dll file isn't even getting loaded. It's supposed to be loaded transitively through the system opengl32.dll, but maybe some systems don't import that. Do you use any anti-virus software, and if so, what?
I'll look into some other loading methods when I get the chance. Thanks!

I use windows 10's own built in anti virus - Defender.   No detections or threats detected.
I was surprised launching EarthVR from coomand line spewed osvr server stuff but nothing really helpful for us, i don't think.
`Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
PS C:\Users\Tron> cd E:
PS E:> cd .\SteamLibrary\steamapps\common\EarthVR
PS E:\SteamLibrary\steamapps\common\EarthVR> .\Earth.exe
PS E:\SteamLibrary\steamapps\common\EarthVR>
VR Server (v1479163853)
Using breakpad crash handler
Setting breakpad minidump AppID = 250820
Forcing breakpad minidump interfaces to load
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
Nov 20 22:46:20.463 info [OSVR]: 00007FFECFC822F8C:\Program Files (x86)\Steam\steamapps\common\SteamVR\bin\win64\vrserve
r.exe
Nov 20 22:46:20.463 info [OSVR: org.osvr.SteamVR]: OSVR client context initialized for org.osvr.SteamVR
Nov 20 22:46:20.467 info [OSVR: org.osvr.SteamVR]: 00007FFECFEF4318
Nov 20 22:46:20.540 info [OSVR: org.osvr.SteamVR]: Connected 0 of 0 unconnected paths successfully
Nov 20 22:46:20.541 info [OSVR: org.osvr.SteamVR]: Connection process took 74ms: have connection to server, have path tr
ee
Nov 20 22:46:20.541 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Commo
n\ClientInterface.cpp:38: Interface initialized for /me/head
Nov 20 22:46:20.542 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Clien
t\TrackerRemoteFactory.cpp:93: Constructed a TrackerHandler for org_osvr_filter_videoimufusion/HeadFusion@localhost:3883
sensor 0
Nov 20 22:46:20.542 info [OSVR: org.osvr.SteamVR]: Successfully produced handler for /me/head
Nov 20 22:46:20.542 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Commo
n\ClientInterface.cpp:38: Interface initialized for /me/head
Nov 20 22:46:20.542 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Commo
n\ClientInterface.cpp:38: Interface initialized for /me/head
Nov 20 22:46:20.542 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Clien
t\DisplayConfig.cpp:195: Display: OSVR HDK (Version 1.3) [Display descriptor note: Specific to the optics of 1.3, with a
specialized distortion correction requiring apps to use RenderManager 0.6.40 or newer.]
Nov 20 22:46:20.542 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Clien
tKit\DisplayC.cpp:47: Created an OSVR_DisplayConfigObject!
Nov 20 22:46:20.544 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Commo
n\ClientInterface.cpp:38: Interface initialized for /me/head
OSVRDisplayConfiguration::parse(): Using mono point sample distortion.
OSVRDisplayConfiguration::parse(): Processing JSON data into mono point samples description structure.
OSVRDisplayConfiguration::parse(): Initial processing complete. Loaded mono point samples data with 1411 and 1411 sample
s per eye, respectively.
OSVRDisplayConfiguration::parse(): Using distortion method "mono_point_samples_built_in": "OSVR_HDK_13_V1"
OSVRDisplayConfiguration::parse(): Using mono point sample distortion.
OSVRDisplayConfiguration::parse(): Processing JSON data into mono point samples description structure.
OSVRDisplayConfiguration::parse(): Initial processing complete. Loaded mono point samples data with 1411 and 1411 sample
s per eye, respectively.
OSVRDisplayConfiguration::parse(): Using distortion method "mono_point_samples_built_in": "OSVR_HDK_13_V1"
OSVRDisplayConfiguration::parse(): Using mono point sample distortion.
OSVRDisplayConfiguration::parse(): Processing JSON data into mono point samples description structure.
OSVRDisplayConfiguration::parse(): Initial processing complete. Loaded mono point samples data with 1411 and 1411 sample
s per eye, respectively.
OSVRDisplayConfiguration::parse(): Using distortion method "mono_point_samples_built_in": "OSVR_HDK_13_V1"
Nov 20 22:46:20.674 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Commo
n\ClientInterface.cpp:38: Interface initialized for /org_osvr_filter_videoimufusion/HeadFusion/semantic/camera
Nov 20 22:46:20.674 info [OSVR]: Z:\j\workspace\OSVR-Core-Build-Win\BIT\64\TRACING\OFF\VS\12\host\windows\src\osvr\Clien
t\TrackerRemoteFactory.cpp:93: Constructed a TrackerHandler for /org_osvr_filter_videoimufusion/HeadFusion@localhost:3883
sensor 2
Nov 20 22:46:20.675 info [OSVR: org.osvr.SteamVR]: Successfully produced handler for /org_osvr_filter_videoimufusion/HeadFusion/semantic/camera
`
Yes.   Enjoy the rest of your Sunday.  :-)

Here's what I have in my log.
[2016-11-21 22:28:25.345] [FakeVive] [info] FakeVive loaded into C:\Program Files (x86)\Steam\steamapps\common\EarthVR\Earth.exe [2016-11-21 22:28:25.346] [FakeVive] [info] Command line: "C:\Program Files (x86)\Steam\steamapps\common\EarthVR\Earth.exe" -FakeViveDebug [2016-11-21 22:28:25.346] [FakeVive] [debug] Using system ddraw.dll at C:\Windows\system32\ddraw.dll [2016-11-21 22:28:25.349] [FakeVive] [info] Installing VR_GetGenericInterface hook [2016-11-21 22:28:25.368] [FakeVive] [info] VR_GetGenericInterface hook installed
That's all. And the HMDNotFound error still appear. Btw, even I used the Hex method, it's not working, still HMDNotFound error @@

@Shockfire There are no integrity checks on OpenVR DLL, you could use that as an entry point and either load the original DLL from the SteamVR folder or ship a differently named version.
Or you could patch the IAT of the OpenVR DLL to load your DLL as a dependency.

Hey again, Can confirm the patched IAT works with regard to getting to the next hardware device check - which is stereo audio device.

FakeVive option now gives verbose debugging in coomand prompt window and there is a fakevive.log made in users appdata local temp dir.

@Balderick You... you don't have a stereo audio device?

I just tried the patched IAT and I also get the error about "Stereo Audio" & it doesn't start :( & yes I have Stereo Audio :P

Hey everyone,
I've developed a method for injecting the DLL which involves using a launcher program. Not sure why the modified openvr_api.dll is causing problems, but this should hopefully work pretty reliably. Try it out and let me know if it works.
How to use this:

Restore your openvr_api.dll to the original version and delete ddraw.dll.
Unzip the attached zip into your EarthVR folder. You should see a FakeVive.dll and a FakeViveLauncher.exe.
Run FakeViveLauncher.exe. If it works, you can add this to Steam as a non-Steam game to make using it easier.

(also, apologies for not responding sooner, I've been pretty busy with other obligations)
FakeViveLauncher.zip

Just tried it & can confirm it's working great :D Thanks!

I tried the FakeViveLauncher.exe, but when I run that, I get the message in the attached screen shot.

I have received the persistent error no matter what I try:
"OpenVR failed to initialize with the given error: VRInitError_Init_HmdNotFoundPresenceFailed"
I am not sure if this is a faulty install, a result of installing Earth VR before installing SteamVR, or something else.
I have tried v.1 and v1.2 of FakeVive, as well as the FakeViveLauncher, and while -FakeViveDebug works fine and returns a log, there isn't an error in it.  Any ideas what might be wrong?

OK, also have a problem with damn thing not running.
Getting following message right after installing Google Earth VR (with or without FakeVive) :

Runned 1.2 with log option, on the screen I get:

[2016-12-27 21:07:10.298] [FakeVive] [info] FakeVive loaded into D:\Program Files (x86)\SteamLibrary\steamapps\common\EarthVR\Earth.exe
[2016-12-27 21:07:10.300] [FakeVive] [info] Command line: "D:\Program Files (x86)\SteamLibrary\steamapps\common\EarthVR\Earth.exe" -FakeViveDebug
[2016-12-27 21:07:10.300] [FakeVive] [debug] Using system ddraw.dll at C:\WINDOWS\system32\ddraw.dll
[2016-12-27 21:07:10.302] [FakeVive] [info] Installing VR_GetGenericInterface hook
[2016-12-27 21:07:10.325] [FakeVive] [info] VR_GetGenericInterface hook installed

(same in the log located in Temp folder)
Windows 10 Pro + Classic Shell, x64 , Intel i7 , 16GB RAM , NVIDIA GTX 1070 .

April 2017 update : Earth stopped working a few days ago, after I upgraded my CG. But there was also updates both of Oculus & Steam VR since my last launch so I can't pinpoint the faulty one.
ddraw.dll seems not to be loaded anymore, and my old patched executables works but without controllers. After a few searches I found this topic, and can confirm that FakeViveLauncher.exe still works.
Thanks again!

I was using the dll only. This is not working anymore and I can not find this one number in the exe using the hexeditor - they changed something. Now I deleted the dll and I am using the FakeViveLauncher. This is working and I do not have a modified Earth.exe in the folder.

The same happendend. Updated Earth, dll not even loading.
I don't find the ViveLauncher.exe you're talking about, where can I find it ?
Thanks

FakeVive its called. Take this zip.

And... new version is there - with Oculus Support. In Steam and directly in the Oculus Store.

Getting the following error when launching earth.exe or from steam.
"OpenVR failed to initialize with the given error: VRInitError_Init_PathRegistryNotFound"
I've tried both ddraw.dll and FakeVive.dll with FakeViveLauncher.exe

As I said... there is a new version - with official Oculus Support. Just deinstall the old one and install the new one. You don't need FakeVive or anything else.


Set IP address

How can I set the IP address so that it does not change when the Internet connection is lost?

If you ISP change your wan ip you may consider using DynDNS

Thanks, I'll give it a try

DynDNS makes it easier to address your node in remote Wallets like ZAP, but it will still create a changing IP in your 1ML history. To have a stable IP/address to the rest of the lightnig network with the RaspiBlitz there are at the moment two ways:

TOR - switching to TOR will give you stable onion-address and tell that to the Lightning Network that will stay the same, even if your ISP gives you a new IP.

SSH-Reverse Tunnel - the RaspiBlitz has a experimental feature that you can use a server with a stable IP on the internet as address proxy and get the traffic from some specific ports forwarded to your RaspiBlitz at home. That ways your IP address to the LN stays stable even if your ISP changes your local IP. For details see: https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#how-to-setup-port-forwarding-with-a-ssh-tunnel

Please add further comments .. but I will close this issue for now.


Enhance the Readme please :-)

Hi,
i want to try your converter. But I can´t figure out how to use it. Could you add a small HowTo into the Readme of your Repo?
Best regards
Christian Dülsen

Hi @DUELCHR , sorry for the delay.
This project is in a very previous version, I'm also thinking in a start it again.
But if you want to try, implement the BAdI ZCONFLU_EXPORT to define the connection

Then just run
    " Create the instance with the Confluence space key and the package to export
    DATA(confluence) = zcl_conflu_obj_documentation=>create( space_key = `MSO` package = 'ZHIGHQ' ).

    " Apply object filter
    confluence->export( VALUE #( ( object_type = 'CLAS' object_name = 'ZCX_HQ_API' ) ) ).

    " Run for all package objects
    confluence->export( ).

It only works with classes and interfaces and gets the content from ABAPDoc


[Additional Request] Change tone of texts

This is sent by email from a person who's sister is using the app. Something to look through and consider.
Instead of saying “Private period-tracking app with customizable symptoms, built for privacy-conscious users.” > Say “Take control of your cycle and symptoms - made for conscious women, by women.”
Using a more proactive tone makes it more engaging for the reader. Moreover, using the phrase ‘conscious women’ can resonate well with the target audience.
From “Made for friends by friends” > To “Empowering women, one download at a time.”
Changing the focus from ‘friends by friends’ to ‘empowering women’ shifts the attention from the app creators to the users, showcasing the empowering factor of the app.
Instead of listing “Period and ovulation tracking with statistics”, “Customizable, color-coded symptoms”, “Cycle prediction…”. > Say “Unlock data-driven insights into your cycle. Get a clear picture of your periods, symptoms, and ovulation. Plus, predict your cycles and stay on top of your health.”
Combining these features in one paragraph while using words like ‘unlock’, ‘clear picture’, and ‘stay on top’, makes the content more engaging and easier to consume.
Don’t just say “No tracking: Your data stays on your phone—no data is sent anywhere” > Say “Total control, total freedom: No data is sent anywhere, period. Your cycle, your data - entirely in your hands.”
Rephrasing this point in a more assertive and empowering way can reassure users about the data safety features.
Also consider adding a clear call-to-action at the end of your content to make it more actionable for the readers.
Just a thought: Rephrase and combine some of these features to make the content flow better and be more concise. Prioritize the most impactful and unique selling points of your app and avoid stating any feature without its benefits.
Give users a reason to engage with your content and consider trying out the app.

We are not only women working on this app, so it would be wrong to say "for women by women"
I think we should avoid the word "empowering", its used so much that it has lost meaning. (And me personally am tired of hearing it)
I think using phrase "unlock data-driven insights" might actually turn away some users as it seems like its a lot of statistics and complicated. (also sounds a bit AI-y)
Too long of a slogan. We like the one we have.


ECONNREFUSED on simple blink tutorial

I'm 10 minutes into playing with my new tessel. Seems its already broken. Any help here?
I pushed a previous project with tessel-av which is when the problems started. But now,even when I erase and run the blink project, it crashes after a second. It still says "i'm blinking" but then immediately crashes.
Thanks!
I'm blinking! (Press CTRL + C to stop)
Socket: Error occurred,  { [Error: connect ECONNREFUSED /var/run/tessel/port_a]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '/var/run/tessel/port_a' }
Socket: Error occurred,  { [Error: connect ECONNREFUSED /var/run/tessel/port_b]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '/var/run/tessel/port_b' }
/usr/lib/node/tessel-export.js:218
throw new Error('Socket: The Port socket has closed.');
^
Error: Socket: The Port socket has closed.
at Socket. (/usr/lib/node/tessel-export.js:218:13)
at emitOne (events.js:77:13)
at Socket.emit (events.js:169:7)
at Pipe._onclose (net.js:477:12)

oh no! If you run t2 list, does your T2 show up?

and, assuming yes, can you please run t2 version and node v and paste the output?

It shows up, but only on LAN now:
INFO Searching for nearby Tessels...
LAN	pk
--- FROM t2 version ---
INFO Looking for your Tessel...
INFO Connected to pk.
INFO Tessel [pk] CLI version: 0.0.27
INFO Tessel [pk] Firmware version: 0.0.14
INFO Tessel [pk] Node version: 4.4.3
--- from node -v ---
v4.4.5

Appears to be the same issue as https://forums.tessel.io/t/econnrefused-var-run-tessel-port-a/2321, which never got fully resolved – but if you unplug & plug it in again, it should run normally
cc @johnnyman727 @rwaldron

yeah that actually worked, thanks

This will happen if a previous execution left a port open and new execution attempts to use that port. Either way, it's not a t2-cli bug, so closing.


Github action does not recognize the Directory.Build.props

According to the error messages for this part:
- name: Build
run: dotnet build --configuration Release`

in .github/workflows/dotnet.yml does not recognize Directory.Build.props since I have no errors on my ide but bunch of errors with workflow.
Anyone have idea to solve it?

I've tried to update dotnet.yml several times, then I added this run: dotnet build --configuration Release /p:Nullable=enable /p:ImplicitUsings=enable /p:LangVersion=latest to solve it