Using Bun to run arbitrary Javascript code from Elixir

For the LiveSvelte project I'm currently using nodejs to SSR Svelte from within Elixir.
The api for this is something like (you can see the code here https://github.com/woutdp/live_svelte/blob/master/lib/ssr/node_js.ex):
NodeJS.call!({"server", "render"}, [name, props, slots], binary: true)

"server" is the name of the server.js file which defines how to render something.
"render" is the function name
[name, props, slots] are my function arguments
binary handles unicode characters properly
I would like to do something similar with Bun. The API doesn't need to be the same.
I've looked into bun.run/2 but it seems it only allows me to specify a file or command to run, which would not be sufficient as I need to also specify a function and function arguments.
Am I overlooking something and is this something that's already supported? If not, have you considered adding it and if not, would you be open for a PR to add this?
Also I'm noticing that your package uses ports to call to executables. nodejs uses a supervisor which starts a node process. That might be a bit faster than using ports but I haven't benchmarked it.
Also if this is not in the scope of the package please let me know :)

This package is meant to provide a Bun installer with the same spirit of Esbuild.
Yours is an interesting idea, but it falls out of the scope of this project.

Hi @crbelaus thank you for the quick reply! Understood, great project btw!


Set button type explicitly to not submit

Coverage decreased (-0.07%) to 84.59% when pulling 3e089806699245e9f60d1e1864bc7e9018e51839 on eatonphil:submit-button-only-form-submission into 5c810868d5789a45396cb7e815a1ab6d17a64666 on linode:master.

Coverage decreased (-0.07%) to 84.59% when pulling 3e089806699245e9f60d1e1864bc7e9018e51839 on eatonphil:submit-button-only-form-submission into 5c810868d5789a45396cb7e815a1ab6d17a64666 on linode:master.

Coverage remained the same at 84.656% when pulling 7baf12e0fad22621796cfe230ac8bc0d0a8a939d on eatonphil:submit-button-only-form-submission into 5c810868d5789a45396cb7e815a1ab6d17a64666 on linode:master.


Select - rename tokens

Description / User Story
After changing the structure of the component tokens within #239 , we now need to adjust the token names for the Select.

Acceptance Criteria

[ ] Design: All tokens used in the Select are following the new token naming
[ ] Dev: All tokens used in the Select are following the new token naming

Background information

Link to new token naming documentation
#357 should be done for this component in parallel

Discussed with @ChristianHoffmannS2:  No further tasks


fix(check): Handle aliases better in do

By checking the arguments before the flat_map call we can get more
information when we check it which prevents some type errors.
(https://xnning.github.io/papers/let-arguments-go-first.pdf motivates this
but I have yet to read it, will read it later to see if normal function
application should be changed as well)
Fixes #766

bors r+


getSpecificOrientation is not a function in latest master

I've installed latest master and seems like something wrong with getSpecificOrientation. I seen PR where it is bridged, but it is not working for me

@inferusvv is this happening on iOS or Android?

Android. I see, that functionality was reverted in some reasons

Same here.

@inferusvv & @PimDeWitte, getSpecificOrientation is not a method implemented in the Android side of things. Currently only works in iOS. Will make a note to add a blurb about this to the README. PR's for this functionality are welcome.

@andrerfneves Could you tell please why implemented functionality was reverted?

@inferusvv I'm not aware of why the code was reverted. I'm guessing it didn't work as intended after merged. If you would like to read upon that PR/code and bring back the functionality for Android we can open a new PR for it.


log level default value

I didn't find possibility to set default value for log level (to be used when baresip starts) in config file. Is there some way to do it?

the default log level is INFO.
the log level can be set to DEBUG if the -v option is given to baresip executable.
also baresip has commands to toggling the loglevel:
  /loglevel           v        Log level toggle

currently the log level is not possible to set in config file. but this is of course
possible to add. this could be added to the "core" section and look something
like this:
log_level    info

OK, I'll try to do add it when I have extra time.


/eth1/v1/config/fork_schedule returns incorrect data for testnet

Description
The output of /eth/v1/config/fork_schedule is missing entries if there are multiple entries at the same epoch.
Steps to Reproduce (Bug)
On zhejiang:
curl -s http:/localhost:xxxx/eth/v1/config/fork_schedule | jq .
{
  "data": [
    {
      "previous_version": "0x00000071",
      "current_version": "0x00000071",
      "epoch": "0"
    },
    {
      "previous_version": "0x00000071",
      "current_version": "0x00000072",
      "epoch": "1350"
    }
  ]
}

Running the same against a different beacon node:
curl -s http:/localhost:xxxx/eth/v1/config/fork_schedule | jq .
{
  "data": [
    {
      "previous_version": "0x00000069",
      "current_version": "0x00000069",
      "epoch": "0"
    },
    {
      "previous_version": "0x00000069",
      "current_version": "0x00000070",
      "epoch": "0"
    },
    {
      "previous_version": "0x00000070",
      "current_version": "0x00000071",
      "epoch": "0"
    },
    {
      "previous_version": "0x00000071",
      "current_version": "0x00000072",
      "epoch": "1350"
    }
  ]
}

Versions (Add all that apply)
teku/v23.1.0+8-gd1cb93d/linux-x86_64/-ubuntu-openjdk64bitservervm-java-11

Thanks Jim.
This is basically the same as #6648, will close the internally raised ticket and take a quick look.


build narratives from dependency parse info

Add a "dependencies to semantic roles" translation mapping. then we could make relatio available in any language with this mapping.

Implemented in the branch relatio-v0.3 (method self.extract_svos() of the NarrativeModel()).
The function for French is in utils.py.
It could be useful to write similar functions for English and German.


CLI: Unable to save to .BIM after loading from TMDL folder

Describe the bug
I am able to load a model from a TMDL folder using the CLI but when I try to save to a .BIM file, I receive the error "File not found: {location of TMDL_Model folder}".
To Reproduce
Steps to reproduce the behavior:

Open PBIX
Open TE2 external tool
Save to folder with TMDL option selected, name folder TMDL_Model
Open command line

Running
start /wait /d "C:\Program Files (x86)\Tabular Editor" TabularEditor.exe "C:\...\TMDL_Model"
successfully opens the model in TE2
while running
start /wait /d "C:\Program Files (x86)\Tabular Editor" TabularEditor.exe "C:\...\TMDL_Model" -B "C:\...\Model.bim"
fails with "File not found: C:\...\TMDL_Model"
Expected behavior
Saving to .bim from a TMSL folder takes similar arguments as above and succeeds so I expected the above to work, particularly since I am able to successfully open the model from the TMDL folder.
Application specifics

Tabular Editor 2.18.8503.15415

Thanks for reporting. This will be fixed in the next update.

This is now fixed in 2.20.0! Thanks for reporting!


Added date picker input field validation to have input as ISO string

Description
Added date picker input field validation to accept input as ISO string
Fixes #3297
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

https://github.com/appsmithorg/appsmith/issues/3297#issuecomment-790595073

@vicky-primathon

Please resolve the merge conflicts
Please update the validation message to specify the use of ISO8601 string | null instead of string | null.


surface: drop wlr_surface_state.buffer_resource

Instead, use wlr_surface_state.buffer only.
Depends on: https://github.com/swaywm/wlroots/pull/3104
Depends on: https://github.com/swaywm/wlroots/pull/3103

LGTM, thanks!


Taint rift? causes unbearable amounts of lag

I got a message saying something about flux that completed the flux research and something spawned on the roof. it does not render anything but looking at it causes the FPS to drop to 2. If i try to place blocks near it they break.

it seems restarting the world fixed it. it now renders and does not cause lag. what could of caused the problem?


Cairo makie

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (eea69b5) 81.88% compared to head (b058810) 81.88%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master       #8   +/-   ##
=======================================
  Coverage   81.88%   81.88%
=======================================
  Files           9        9
  Lines         657      657
=======================================
  Hits          538      538
  Misses        119      119

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Why we have different base images ?

Hi,
First of all thanks for the docker settings.
But may I know why we have different base images ?
Apache - Ubuntu 16:04
Phpfpm & mysql - Debian Wheezy
Workspace - Phusion base image
I am pretty new to docker. Sorry if I am wrong .
Can any one please explain ?
Thanks in advance.

There's no need to have them on the same base image. We choose the best base image for every software. Base images have different sizes and pre-installed software's, so they could be required by some software's and not by others..

For apache specifically, wouldn't it make sense to use debian-7 (wheezy) as the base image? See here, they have builds with debian. http://dockerfile.readthedocs.io/en/latest/content/DockerImages/dockerfiles/apache.html


DM-37933: Overhaul tag parsing

Move StandaloneRSPTag to a new RSPImageTag class in a separate domain model that provides an abstract data type. The rest of the current models/tag.py code will move in a separate change. Rework the parsing code to hopefully be easier to follow and a bit more succinct.
Comparing incomparable tag types now behaves like comparing any other incomparable types.  The cost is a less specific and verbose error message, but given that this indicates a programming error, that seems fine for the additional simplicity.

And I don't know if this belongs in a common package, or something that is explicitly a SQR-059 implementation.  It may be that nothing outside the Lab Controller ever really needs to care about tag format and meaning, although it will be nice to be able to wave it at our hapless release engineer the next time they manage to come up with an RC Candidate whose tag somehow doesn't match any of the existing and documented formats.

Yes, indeed, there are follow-up diffs to also move RSPTag that are coming. The goal at the end of this is to create abstract data types for representing RSP images and image tags that we can lift into Safir down the road and reuse in other projects. The starting point will have just the methods required for the lab controller, but the idea is that they won't otherwise depend on any lab controller code. They're just a generic representation of a tag and an image.


I'm sorry but, it doesn't work for me

First of all, thanks for your attention.
I am following the steps of his instructions but the pdf that is a single page does not load, only the window appears with "Previous Next Page: /" I have tried with firefox chrome and google chrome movile ..
I have also tried to load other pdf files with more than one page but I have the same result.
I don't know if I'm doing something wrong (I'm just starting with hugo), I'm trying to make it work at https://hotspotcepeda.github.io/es/posts/hotspot/#pdf
I have the pdf file uploaded locally at https://github.com/hotspotcepeda/hotspotcepeda.github.io/blob/main/static/pdfs/hotspot_conectar.pdf
I would like to know what is happening, I do not have an error message, the pdf file is simply not displayed.
I would appreciate your help in making it work.

Thanks for raising the issue.
I am a little busy with my full time job this week.
i will try to spend time when I can and check this.

I can get it to work by changing this line at the top of embed-pdf.html from this:
<script type="text/javascript" src="/js/pdf-js/build/pdf.js"></script>
to this:
<script type="text/javascript" src="/my-project-name/js/pdf-js/build/pdf.js"></script>
Problem is, that's not really a "fix" so much as a "hack to make it work on my project".

<script type="text/javascript" src= "{{"/" | relURL}}/js/pdf-js/build/pdf.js"></script> seems to be working for me.  Merge request via fork?  I'm not sure what kind of tests you have going here.

<script type="text/javascript" src= "{{"/" | relURL}}/js/pdf-js/build/pdf.js"></script> seems to be working for me. Merge request via fork? I'm not sure what kind of tests you have going here.

<script type="text/javascript" src= "{{"/" | relURL}}/js/pdf-js/build/pdf.js"></script> seems to be working for me.
Awesome!

Merge request via fork?

Yes, Please!

I'm not sure what kind of tests you have going here.

ermm. none .
I wrote the shortcode for a work related (open-source) documentation website and realised no shortcode exists.
So thought of making it into one.
I don't think I know how to setup tests on this kind of a project.
If you know how this could be done, would appreciate it if you could help .

Not sure why the issue was closed, let me get a PR going.  I can walk you through how we're deploying hugo-- you can see it in this project here: https://github.com/tetrabiodistributed/papra  We're basically deploying hugo through a docker image and through build actions to get everything compiled, with this project as a submodule.  I don't know how well this fix will work for other people, so I'd love some confirmation that it doesn't break what they're up to.

Thank you @mmroden I have a demo site up with the shortcode and I am deploying it using Netlify.
The demo site is here and the source code is here
May be I could set up the shortcode as a submodule in the demo site and deploy every PR. But getting that integration on this repository will be difficult, I am assuming.

@mmroden I think the OP closed the issue since you provided a fix.
Thanks for opening it again.


Fix example.py

This particular item for some reason gives a 403 error, works for any other.

Thank you for your contribution! I changed the target branch to develop.


Gentrification Page

On the Transform page there's a section about Gentrification. Add a link that leads to a whole page going more deeply on the topic.

Here's a link to the prototype where you should be able to copy the text.

I'd like to be assigned this please.

closed by #552, #553


issue with stopping task with id

[x] add a task stop --all command to stop all datmo tasks
[x] fix environment run execution to ensure that the run_id is set in the task. this will ensure stop for task will always work

working off of the flow-issues branch

the second point was not possible, because the run may not even complete and may not be returned by docker. instead, stop tasks uses the task name which is given at create time to the environment driver. once it is stopped (and the running container removed), the task will have a status=STOPPED
commit e6347ec  has these changes

based on this we also closed #109 since it isn't necessary nor possible at the moment.

fixed with PR #115


Index out of range for CSS Select

I am getting the following error when running the example script from the project site (here: https://github.com/karan/TPB#usage):
File "example.py", line 34, in
print(torrent.info)
File "/usr/local/lib/python2.7/dist-packages/ThePirateBay-v1.3.5-py2.7.egg/tpb/tpb.py", line 347, in info
info = root.cssselect('#details > .nfo > pre')[0].text_content()
IndexError: list index out of range
The only change I made, was replacing the domain from "https://thepiratebay.org" to "https://thepiratebay.la"

I made a duplicate without noticing...

i'm getting the same error could you provide any more details about where the duplicate was?

To be honest, I have no idea. At the time of writing it was a duplicate, but I don't know if the issue was resolved, closed or deleted. Sorry!


Log govuk request id in json in whitehall

Change Rails logger to log in JSON format including govuk_request_id for traceability.
Remove the $real_stdout, $stdout and $stderr redirection because LOGIT is able to consume not JSON logs now.
Related Trello card: https://trello.com/c/cfTa0MXn/58-story-log-govukrequestid-from-railslogger-in-whitehall-and-its-worker

Just a heads up that I'm out at a conference tomorrow (Friday) and on leave Monday, but you shouldn't need to wait for me to approve as I think you're just about there - though looks like some WIP stuff is in at the mo so didn't want to approve.
I did notice that two of the conversations I've opened have been marked as resolved without the changes applied so assume there was either a communication issue, a mistake or still work-in-progress. I'll just highlight them for clarity:
https://github.com/alphagov/govuk_app_config/pull/300#discussion_r1246505266 requesting: https://github.com/alphagov/govuk_app_config/pull/300/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4edR3 to be changed to * Add govuk_request_id to JSON logging for apps with gds-api-adapters ([#300](https://github.com/alphagov/govuk_app_config/pull/300))
https://github.com/alphagov/govuk_app_config/pull/300#discussion_r1246505728 requesting the deletion of: https://github.com/alphagov/govuk_app_config/pull/300/files#diff-45d9c2f711166ef24efefacb50998affe202ae140a2385005721b33d8af8ab83R141-R145 since we don't need it anymore as the class is stubbed in the test.

Just a heads up that I'm out at a conference tomorrow (Friday) and on leave Monday, but you shouldn't need to wait for me to approve as I think you're just about there - though looks like some WIP stuff is in at the mo so didn't want to approve.
I did notice that two of the conversations I've opened have been marked as resolved without the changes applied so assume there was either a communication issue, a mistake or still work-in-progress. I'll just highlight them for clarity:
#300 (comment) requesting: https://github.com/alphagov/govuk_app_config/pull/300/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4edR3 to be changed to * Add govuk_request_id to JSON logging for apps with gds-api-adapters ([#300](https://github.com/alphagov/govuk_app_config/pull/300))
#300 (comment) requesting the deletion of: https://github.com/alphagov/govuk_app_config/pull/300/files#diff-45d9c2f711166ef24efefacb50998affe202ae140a2385005721b33d8af8ab83R141-R145 since we don't need it anymore as the class is stubbed in the test.

Hey you are right, I missed those two changes and added them back now :D

This looks good, looks you've still got a linting to fix and we noticed in https://github.com/alphagov/govuk_app_config/pull/302 that the readme should be updated: https://github.com/alphagov/govuk_app_config/pull/302#discussion_r1251915399


Release pmgr 0.16

There is one more place where 0.15 is mentioned. It is out.xml file. Please add change of it to this PR.

We'll wait for Taurus to make sure it works for them and release then. @greyfenrir will work on that from Taurus side.


General problem description

⚠ Support requests in an issue-format will be closed immediately. For support, go to Swarm's Discord.
Context
incorrect currency requested during startup
Summary
Expected behavior
should read eg. '...sufficient xDAI (for Gas) and at least 1 BZZ bridged on the xDAI network...'
Actual behavior
this was displayed
Nov 27 13:49:26 ip-172-31-18-31 bee[17534]: time="2021-11-27T13:49:26Z" level=warning msg="cannot continue until there is sufficient ETH (for Gas) and at least 1 BZZ available on 50078a907a97fda390f8a0733005731eefa7244f"

Steps to reproduce
start a node on mainnet with no currency in wallet

IMO the message should be either specific about both gbzz and xdai, or should be chain agnostic


Add BoxSet constructor, fix bad url in docs | latest button, ensure consistent one-based indexing in BoxPartition

I'm preparing the update to FLoops.jl, and have some small changes that are required for that future PR. Also some docstrings had typos.
Adds a constructor to allow Boxsets to be made using arrays of boxes, ie
S = [Box(center_1, radius_1), Box(center_2, radius_2), Box(center_3, radius_3)]
P = BoxPartition(Box(center_1, radius_1 .+ radius_2 .+ radius_3), (100,100,100))
B = P[S]

returns a covering of S using boxes from P.
Also, some sections in partition_regular.jl use zero-based indexing, which caused some confusing errors. They are now fixed. partition_tree.jl still uses zero-based indexing and one-based indexing together; I'm considering a rework on that file.

Useful indeed.


PowerShell Steps

Could you share the PowerShell Steps as well?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 692d8637-4033-767b-aa29-3859f301b7b8
Version Independent ID: de143a26-3dd6-a513-6854-604e4f68c52f
Content: ActiveDirectoryInteractive connects to SQL
Content Source: articles/sql-database/active-directory-interactive-connect-azure-sql-db.md
Service: sql-database
GitHub Login: @GithubMirek
Microsoft Alias: MirekS

@Ayanmullick Thank you for the detailed inquiry. Are you looking for general Azure Activity Directory PowerShell cmdlets or are you looking for a PowerShell content specific to this document? General Azure Activity Directory PowerShell examples can be found: PowerShell.
This document is specific to a C# based application making a connection to Azure SQL Database with Azure Active Directory. PowerShell is not the focus of this exercise.

I'm looking for the PowerShell equivalent of the C# source code; as it is documented in other SQL docs.
https://docs.microsoft.com/en-us/sql/relational-databases/server-management-objects-smo/tasks/creating-altering-and-removing-databases?view=sql-server-2017

Thanks for the clarification @Ayanmullick. This is being evaluated by the content team and updated accordingly.

Making @CarlRabeler aware.

Making @VanMSFT aware

@VanMSFT please investigate

Following up with @GithubMirek

Hello @Ayanmullick, thank you for your inquiry. I checked with the Product Team on whether they have a PowerShell example for the requested document. Unfortunately, there's no samples available, and no plans at the moment to provide it. We will evaluate adding different types of samples in the future to improve our documents, and we appreciate the feedback. Thanks!

#please-close

@Mike-Ubezzi-MSFT please close


Displaying .raw files not working.

Hi,
I have a problem when using your library. I want to draw custom .raw image on my display and when i am using one from your examles it works flolessly but when i want to use my image it just display some weird random pixels and i dont know what to do with that. In attachment i have placed what images i want to display (I know it has to be raw type but github does not support it). Please help!

edit: afetr looking for it you mentioned  img2rgb565.py but i cannot find it anywhere

ok i found and it works sorry for problem. :)


Just curious: why waiting until completion to open source Gaffer2?

Hi.  I'm just curious why you're waiting until Gaffer2 is completed before publishing it under an open source license.  Is there some reason not to do its development in the open?
(Serious question, not meant facetiously, in case that's not clear.  I can think of both good and bad reasons to wait, and wasn't sure which ones were operative here -- so I thought I'd just ask!)

Hi. Gaffer2 will not be complete when it is published - it will be a work in progress with plenty of room for further development.

Thanks, @gaffer01.  What I was really asking is what is the publication boundary -- that is, given the above, why not do the work in an open repository starting from the beginning?

@kfogel Most likely because the code has to be fully audited before being released, to ensure nothing is being leaked out to the public, and also to protect themselves against negative press (e.g. dodgy commit messages, leaked info etc). Github promotes social development, but this is the exact opposite of how GCHQ work, at least from an external standpoint, instead they are using Github as a release mechanism, rather than a development workflow tool. Shame really.

@gaffer01 I hope you are thinking of ways you can move away from this simplistic publication model, and into a more collaborative mode. I understand it's all happening within an organisation that has thousands of internal barriers to collaboratively working on software in an open community, but without that mutual trust, you haven't built any trust, just gave us a tool, which is fine, but ultimately disappointing and not worthy of the same excitement that most other interesting OS projects command.

Hmm.  There's a lot of speculation in this thread as to the reasons GCHQ isn't doing the work in the open in the first place.  Only @gaffer01 can say for sure which reasons are correct and which aren't, but if it were just a matter of code having to be audited before being published, the usual method for solving that -- a method that continues to work after publication -- is to have a dedicated branch for pre-audit code, and keep that branch only in internal repositories.  As soon as changes are approved, they can go to out to the public repository (on whatever branch is appropriate, presumably master).  Git makes this pretty easy; I have worked on projects that published to two repositories and it was fine.  One can set up hooks ("triggers" I guess git calls them) to make sure changes don't get published in the wrong repository just because someone made a typo when pushing, obviously.
@foxx and @SamMorrowDrums: As to whether GCHQ should be working more collaboratively, that's a separate question from publication, and I don't think we know enough to say whether they should be more collaborative or not.  It's not all upside, after all -- they might have good reasons for choosing one side of that tradeoff instead of the other.
More generally, I think there's often a mistaken impression that "open source" equals "collaborative development methodology".  The two don't always go together, and that's okay.  The only thing open source is definitely about is users' freedom to copy, customize, and redistribute the software as they see fit.  Whether a broadly collaborative development model makes sense for a particular project is a complicated question, and involves many more factors than just the license the software is under.
So I'm not making any assumptions about what reasons GCHQ has for not open-sourcing their code from Day One, nor about whether they should try for a more collaborative style of development.  However, I am emphasizing to @gaffer01 that one can put stuff in a public place starting from the earliest stages of development, without being under any obligation to change one's development methodology by taking on more collaborators.  While the latter would require the former, the former does not force the latter.

The Gaffer2 code is currently being tidied and reviewed before release. Once it has been released, further development will be done here. The hope is to be able to accept contributions from the community, while retaining control of the project in the short term.

Thanks, @gaffer01.  Looking forward to seeing it.


fix: setting ttl to 0 to remove jobs on completion

I am expecting them too be removed according to k8s docs, and I am not in the business of testing their code. We should file a bug if what we follow is not correct.
What does your test file look like?

Also, that was added in 1.12. Can you investigate wether AWS has that enabled? I expect that to be GA by now, so I assumed it to be available. It must be, otherwise the controller would bork.

You are right, can take a lot of time before this passes as beta or ga. Closing this PR.


Want to Add 3d Safari Logo

Feature Add
want to 3d safari logo.

Thanks for assigning this to me. I will work on it Today.

I have a pr regarding this issue #315. So please if I would be allowed to merge it or not. And how it is any suggestions are welcomed. @VidhiBhatt01


Checkpoint is not work properly

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 x64 1909 Home
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): pip
TensorFlow version (use command below): 2.1.0
Python version: 3.7.6
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
tf.save_weights is not work properly
https://www.tensorflow.org/tutorials/keras/save_and_load#saved_model을_사용하기
When I follow the tutorial of tensorflow checkpoint section, It does not make *.ckpt file but makes .ckpt folder!!!!
Describe the expected behavior
It need make .ckpt files.
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import cv2
import random
import os

np.random.seed(1)
random.seed(1)
tf.random.set_seed(1)

checkpoint_path = 'checkpoint\\cp-{epoch:04d}.ckpt'
checkpoint_dir = os.path.dirname(checkpoint_path)

(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()

train_images = train_images / 255.0
test_images = test_images / 255.0

train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))

model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dropout(0.02, seed=1),
    tf.keras.layers.Dense(254, activation='relu'),
    tf.keras.layers.Dropout(0.02, seed=1),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.01, seed=1),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(32, activation='sigmoid'),
    tf.keras.layers.Dense(10)
])

checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    verbose=1,
    save_weight_only=True,
    period=2
)

model.save_weights(checkpoint_path.format(epoch=0))

epochs = 10

model.compile(
    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy']
)

model.fit(
    train_dataset.shuffle(int(len(train_images)+500), seed=1).batch(256),
    epochs=epochs,
    callbacks=[checkpoint_callback],
    verbose=1
)

loss, acc = model.evaluate(test_images,  test_labels, verbose=2)
print("Untrained model, accuracy: {:5.2f}%".format(100*acc))

@shwoghk14
Please share simple stand alone code for us to replicate the issue faced, or share a colab gist with the error for us to analyse the issue faced

@shwoghk14
Please share simple stand alone code for us to replicate the issue faced, or share a colab gist with the error for us to analyse the issue faced

I was added code that take weird actions.
Thanks.

@shwoghk14
I ran the code shared , please refer to this gist and let us know if it confirms your issue

@shwoghk14
I ran the code shared , please refer to this gist and let us know if it confirms your issue

@Saduf2019
Yes! If you seed the /content where saving location of checkpoint, you can see the folders but not .ckpt files.
It's odds!

@shwoghk14 I have changed your code very little and it is working as expected. Please verify once and let me know if this is what you are expecting. Thanks!
Please close the issue if this was resolved for you. Thanks!

@jvishnuvardhan Where can I see the code which was eidted?
Thanks.

@shwoghk14 Sorry, forgot to attach. Here is the gist. Thanks!

@shwoghk14 Sorry, forgot to attach. Here is the gist.
I ran with TF2.1 and tf-nightly. You can run with TF2.1, don't need to change (but suggested as newer versions are has better performance). Thanks!

@jvishnuvardhan I ran my computer but it occur a error.
tensorflow.python.framework.errors_impl.NotFoundError: Failed to create a directory: checkpoint/cp-0001.ckpt\variables; No such file or directory

It is why I instead use \ but / in  checkpoint_path.
Anyway I ran your code on gist and on my local computer(use \ in path) not changed behaviors.
It makes folders not files.
tensorflow.python.framework.errors_impl.NotFoundError: Failed to create a directory: checkpoint/cp-0001.ckpt\variables; No such file or directory

I will check on my windows laptop and let you know the solution. Thanks!

@jvishnuvardhan
Is it not done? The tensorflow bot says this issue is closed soon.

@shwoghk14 bot will not close this. I will check it on my windows laptop later today. Thanks!

@shwoghk14 I ran it in my Windows10 laptop. It works without any error. Please check the log.
Please verify once and close the issue if this was resolved for you. Thanks!
C:\Users\XXXXX>python C:\Users\XXXXX\Downloads\GitHub_models\checkpoints.py
2020-07-07 17:25:22.093668: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-07-07 17:25:22.103596: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1dff5494650 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-07 17:25:22.109377: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-07 17:25:23.270684: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 376320000 exceeds 10% of free system memory.
2020-07-07 17:25:23.662429: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 376320000 exceeds 10% of free system memory.
Epoch 1/10
234/235 [============================>.] - ETA: 0s - loss: 0.3782 - accuracy: 0.9316
Epoch 00001: saving model to checkpoint/cp-0001.ckpt
2020-07-07 17:25:29.053866: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
WARNING:tensorflow:From c:\users\XXXXX\appdata\local\programs\python\python36\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
235/235 [==============================] - 5s 21ms/step - loss: 0.3778 - accuracy: 0.9316
2020-07-07 17:25:30.085141: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 376320000 exceeds 10% of free system memory.
Epoch 2/10
234/235 [============================>.] - ETA: 0s - loss: 0.1064 - accuracy: 0.9751
Epoch 00002: saving model to checkpoint/cp-0002.ckpt
235/235 [==============================] - 5s 21ms/step - loss: 0.1064 - accuracy: 0.9751
2020-07-07 17:25:35.396674: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 376320000 exceeds 10% of free system memory.
Epoch 3/10
234/235 [============================>.] - ETA: 0s - loss: 0.0686 - accuracy: 0.9824
Epoch 00003: saving model to checkpoint/cp-0003.ckpt
235/235 [==============================] - 5s 21ms/step - loss: 0.0686 - accuracy: 0.9824
2020-07-07 17:25:40.746641: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 376320000 exceeds 10% of free system memory.
Epoch 4/10
233/235 [============================>.] - ETA: 0s - loss: 0.0492 - accuracy: 0.9870
Epoch 00004: saving model to checkpoint/cp-0004.ckpt
235/235 [==============================] - 5s 21ms/step - loss: 0.0494 - accuracy: 0.9870
Epoch 5/10
234/235 [============================>.] - ETA: 0s - loss: 0.0373 - accuracy: 0.9898
Epoch 00005: saving model to checkpoint/cp-0005.ckpt
235/235 [==============================] - 5s 22ms/step - loss: 0.0374 - accuracy: 0.9898
Epoch 6/10
235/235 [==============================] - ETA: 0s - loss: 0.0318 - accuracy: 0.9912
Epoch 00006: saving model to checkpoint/cp-0006.ckpt
235/235 [==============================] - 5s 23ms/step - loss: 0.0318 - accuracy: 0.9912
Epoch 7/10
234/235 [============================>.] - ETA: 0s - loss: 0.0257 - accuracy: 0.9928
Epoch 00007: saving model to checkpoint/cp-0007.ckpt
235/235 [==============================] - 5s 22ms/step - loss: 0.0257 - accuracy: 0.9928
Epoch 8/10
234/235 [============================>.] - ETA: 0s - loss: 0.0238 - accuracy: 0.9936
Epoch 00008: saving model to checkpoint/cp-0008.ckpt
235/235 [==============================] - 5s 22ms/step - loss: 0.0238 - accuracy: 0.9937
Epoch 9/10
235/235 [==============================] - ETA: 0s - loss: 0.0212 - accuracy: 0.9944
Epoch 00009: saving model to checkpoint/cp-0009.ckpt
235/235 [==============================] - 5s 21ms/step - loss: 0.0212 - accuracy: 0.9944
Epoch 10/10
233/235 [============================>.] - ETA: 0s - loss: 0.0179 - accuracy: 0.9951
Epoch 00010: saving model to checkpoint/cp-0010.ckpt
235/235 [==============================] - 5s 22ms/step - loss: 0.0181 - accuracy: 0.9951
313/313 - 1s - loss: 0.0891 - accuracy: 0.9787
Untrained model, accuracy: 97.87%

@janapativishnu Hello.
The error is not a point.
My issue is the abnormal behavior of checkpoint.
I has known the checkpoint makes .ckpt file.
But my code makes .ckpt folder.
That is a point.

@shwoghk14 It is expected. Code makes .ckpt folder and inside the folder there will be three things (saved_model.pb, and two folders assets and variables).
I used this path (instead of your's)
#checkpoint_path = 'checkpoint\cp-{epoch:04d}.ckpt'
checkpoint_path = '/checkpoint/cp-{epoch:04d}.ckpt'
checkpoint_dir = os.path.dirname(checkpoint_path)
Please check the checkpoint directory.
![GitHub](https://user-images.githubusercontent.com/15189960/86864306-c9f7dd00-c081-11ea-8b6f-eb0a4e502dc6.png)

@shwoghk14 I just tried your path also. It works similar to my path shown in the last comment. Similarly, each of the .ckpt directory has one file and 2 folders. Thanks!

@janapativishnu Yes, I also got that. but, I need a cp-0001.ckpt file, not cp-0001.ckpt folder.
Make a folder is expected behavior?
but I doing this code which on below URL makes cp-0001.ckpt file.
https://www.tensorflow.org/tutorials/keras/save_and_load#saved_model을_사용하기
Thanks.

@janapativishnu I'm sorry. My mistakes to say cp-0001.ckpt file.
The tutorial makes cp-0005.ckpt.index, cp-0005.ckpt.data-00000-of-00002, cp-0005.ckpt.data-00001-of-00002, cp-0005.ckpt.index
I want those files which above four not ckpt folders.
this is tutorial code.
import os

import tensorflow as tf
from tensorflow import keras

print(tf.version.VERSION)

(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()

train_labels = train_labels[:1000]
test_labels = test_labels[:1000]

train_images = train_images[:1000].reshape(-1, 28 * 28) / 255.0
test_images = test_images[:1000].reshape(-1, 28 * 28) / 255.0

# Define a simple sequential model
def create_model():
  model = tf.keras.models.Sequential([
    keras.layers.Dense(512, activation='relu', input_shape=(784,)),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(10)
  ])

  model.compile(optimizer='adam',
                loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
                metrics=['accuracy'])

  return model

# Create a basic model instance
model = create_model()

# Display the model's architecture
model.summary()

# Include the epoch in the file name (uses `str.format`)
checkpoint_path = "training_2/cp-{epoch:04d}.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

# Create a callback that saves the model's weights every 5 epochs
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    verbose=1,
    save_weights_only=True,
    period=5)

# Create a new model instance
model = create_model()

# Save the weights using the `checkpoint_path` format
model.save_weights(checkpoint_path.format(epoch=0))

# Train the model with the new callback
model.fit(train_images,
          train_labels,
          epochs=50,
          callbacks=[cp_callback],
          validation_data=(test_images,test_labels),
          verbose=0)

# This may generate warnings related to saving the state of the optimizer.
# These warnings (and similar warnings throughout this notebook)
# are in place to discourage outdated usage, and can be ignored.

model = create_model()

# Evaluate the model
loss, acc = model.evaluate(test_images,  test_labels, verbose=2)
print("Untrained model, accuracy: {:5.2f}%".format(100*acc))

Thanks.

Change save_weight_only=True, to save_weights_only=True (weights with s) in tf.keras.callbacks.ModelCheckpoint and it should create files instead of folders
Ref : https://abstack.pk/post/tensorflow-create-ckpt-files-instead-of-folders

@abStack-pk  HI,
I hadn't know that has missing letter 's'.
Thanks, I question is solved.
Best regards.

Closing this issue as this was resolved. Thanks!


Parse EPA and EIA data into one table of generator data to be used in the HIFLD Grid

:rocket:

[x] Is your feature request essential for your project?

Describe the workflow you want to enable
I wish I could parse data from the EIA and EPA, and align them, to be able to estimate all relevant information for generators (units) or power plants (collections of units).
Describe your proposed implementation
The EPA's AMPD dataset has hourly samples of (power, heat) for units, which can be used to fit a heat rate curve. The EPA's NEEDS dataset has linear heat-rate estimates as a fall-back. See the EPA Power Sector Modeling Platform v6 documentation for the November 2018 and January 2020 versions.
The EIA data has other information we want, like minimum generation level and more details on generator types (e.g. standalone CT vs. standalone ST vs. units that are part of a combined-cycle plant).
The data are not well aligned between EIA and EPA data sets, because the EIA cares about what can make electricity (generators) while the EPA cares about what can make emissions (boilers and CTs), and they both refer to them as 'units', even though the mapping is not always 1-to-1. There is a project that aims to aid in mapping: https://github.com/USEPA/camd-eia-crosswalk, if we can figure out how to use the information effectively.
There is also an open question of at which level we want to be aggregating units, especially for combined-cycle plants. As an example, the West Phoenix plant in Arizona has 3x CT/ST pairs: 1x (91 MW, 45 MW) + 2x (185 MW, 201 MW). Do we want to try to create: a) one plant, containing all generators, b) two plants, one for the smaller CC pair and one for the combination of the identical larger two, c) three plants, one for each CC pair, using each pair's fitted heat rate curve, d) three plants, one for the smaller CC pair, and two identical ones for the larger CC pairs, where we use data from both to fit a single heat rate curve, e) six individual plants, f) something else?
a) Benefits: simple. Drawbacks: overly simplified, different CC units have different characteristics.
b) Benefits: better resolution than (a). Drawbacks: we have to assume that the two larger CCs are identical based on their nameplate capacities (we could also look at things like operational date, but would have to make some assumptions about how close is close enough).
c) Benefits: better resolution than (b). Drawbacks: less data to fit curves for each of the two larger generators
d) Benefits: likely better data quality than (c). Drawbacks: like (b), we need to make assumptions about which generators are similar enough to be considered identical
e) Benefits: better resolution. Drawbacks: loss of connection between units that are connected in reality, less data to fit heat rate curves.
At a higher level, how do we want to parse through the ~20k generators to decide which units to aggregate?
Hopefully this Issue can serve as the hub for brainstorming approaches and discussing challenges in implementation.

There's a lot of overlapping information between the HIFLD datasets (Power_Plants.csv, Generating_Units.csv) and the EIA Form 860 data (Sheet 2___PlantYyyy for plants, Sheet 3_1_GeneratorYyyyy for units). Looking through my current code for #197, it seems that so far we're looking at the HIFLD data columns:

Power_Plants.csv

status
state
county
latitude
longitude
PLANT_CODE

Generating_Units.csv

status
state
ZIP
type
nameplate capacity
PLANT_CODE

Then it seems we only use the EIA data to look up Pmin values. I don't see power plant status explicitly as a part of the EIA Form 860 data, but for everything else there appears to be equivalent information in the EIA data. I would generally have more trust in the EIA data than HIFLD for generators, so I wonder if it makes sense to use all EIA/EPA data for coming up with our generator data set, and only using the HIFLD data for the transmission network. I think it would be simple enough to accomplish: all of our logic and data structures stay the same, we just need to load a different table to start with, and rename a few columns.

If we start from the EIA Form 860 units, we can map 87% of the total summer capacity to at least some data in the EPA AMPD dataset (haven't looked at how much yet or whether it's good enough to generate a heat rate curve from):
>>> import pandas as pd
>>> from prereise.gather.griddata.hifld import const
>>> from prereise.gather.griddata.hifld.data_access import load
>>>
>>> # Helper function
>>> def floatify(x):
...     try:
...         return float(x.replace(",", ""))
...     except ValueError:
...         if x == " ":
...             return pd.NA
...         else:
...             print(repr(x))
...             return pd.NA
...
>>> # Constants
>>> no_ampd_expected_techs = {
...     "Batteries",
...     "Conventional Hydroelectric",
...     "Flywheels",
...     "Geothermal",
...     "Hydroelectric Pumped Storage",
...     "Nuclear",
...     "Offshore Wind Turbine",
...     "Onshore Wind Turbine",
...     "Solar Photovoltaic",
...     "Solar Thermal with Energy Storage",
...     "Solar Thermal without Energy Storage",
... }
>>>
>>> # Load raw tables
>>> # This one takes a while
>>> epa_ampd = load.get_epa_ampd(const.blob_paths["epa_ampd"])
sys:1: DtypeWarning: Columns (10,12,18,20) have mixed types.Specify dtype option on import or set low_memory=False.
sys:1: DtypeWarning: Columns (10,12,14,16,18,20) have mixed types.Specify dtype option on import or set low_memory=False.
sys:1: DtypeWarning: Columns (3,10,12,18,20) have mixed types.Specify dtype option on import or set low_memory=False.
>>> form_860 = load.get_eia_form_860(const.blob_paths["eia_form860_2019"])
sys:1: DtypeWarning: Columns (15,37,38,42) have mixed types.Specify dtype option on import or set low_memory=False.
>>> crosswalk = pd.read_csv("https://github.com/USEPA/camd-eia-crosswalk/releases/download/v0.2.1/epa_eia_crosswalk.csv")
>>>
>>> # Initial processing of the EPA AMPD
>>> epa_ampd = epa_ampd.astype({"UNITID": "string"})
>>> epa_units = set(epa_ampd.groupby(["ORISPL_CODE", "UNITID"]).groups.keys())
>>> print(len(epa_units))
4313
>>>
>>> # Initial processing of Form 860
>>> form_860 = form_860.astype({"Plant Code": int})
>>> form_860["Summer Capacity (MW)"] = form_860["Summer Capacity (MW)"].map(floatify)
>>> print(len(form_860))
21971
>>>
>>> # Initial processing of EIA/EPA Crosswalk
>>> crosswalk_match_exclude = {"CAMD Unmatched", "Manual CAMD Excluded"}
>>> crosswalk = crosswalk.query(
...     "MATCH_TYPE_GEN not in @crosswalk_match_exclude and CAMD_STATUS != 'RET'"
... ).astype(
...     {
...         "MOD_EIA_PLANT_ID": int,
...         "MOD_CAMD_UNIT_ID": "string",
...         "MOD_CAMD_GENERATOR_ID": "string",
...         "MOD_EIA_GENERATOR_ID_GEN": "string",
...     }
... )
>>> print(len(crosswalk))
5305
>>>
>>> # Build translation Series
>>> crosswalk_translation = crosswalk.set_index(
...     ["MOD_EIA_PLANT_ID", "MOD_CAMD_GENERATOR_ID"]
... ).apply(
...     lambda x: (x["CAMD_PLANT_ID"], x["MOD_CAMD_UNIT_ID"]),
...     axis=1,
... )
>>> print(crosswalk_translation)
MOD_EIA_PLANT_ID  MOD_CAMD_GENERATOR_ID
3                 1                               (3, 1)
                  2                               (3, 2)
                  4                               (3, 4)
                  5                               (3, 5)
                  A1CT                           (3, 6A)
                                               ...
60903             3                           (60903, 1)
                  2                           (60903, 2)
                  4                           (60903, 2)
70454             MAG1                     (70454, MAG1)
                  MAG2                     (70454, MAG2)
Length: 5305, dtype: object
>>>
>>> # Filter translation to just what's in the EPA data
>>> crosswalk_with_ampd_data = crosswalk_translation.loc[crosswalk_translation.isin(epa_units)]
>>> print(crosswalk_with_ampd_data)
MOD_EIA_PLANT_ID  MOD_CAMD_GENERATOR_ID
3                 1                               (3, 1)
                  2                               (3, 2)
                  4                               (3, 4)
                  5                               (3, 5)
                  A1CT                           (3, 6A)
                                               ...
60903             3                           (60903, 1)
                  2                           (60903, 2)
                  4                           (60903, 2)
70454             MAG1                     (70454, MAG1)
                  MAG2                     (70454, MAG2)
Length: 5196, dtype: object
>>>
>>> # Calculate how much of the Form 860 capacity maps to EPA data via crosswalk
>>> form_860_keys = form_860[["Plant Code", "Generator ID"]].apply(tuple, axis=1)
>>> print(
...     "EIA Capacity mapped to CAMD (MW):",
...     "{:,.0f}".format(
...         form_860.loc[form_860_keys.isin(crosswalk_with_ampd_data.index), "Summer Capacity (MW)"].sum()
...     ),
... )
EIA Capacity mapped to CAMD (MW): 651,590
>>> print(
...     "EIA Capacity expected to be mapped to CAMD (MW):",
...     "{:,.0f}".format(
...         form_860.loc[~form_860["Technology"].isin(no_ampd_expected_techs), "Summer Capacity (MW)"].sum()
...     ),
... )
EIA Capacity expected to be mapped to CAMD (MW): 749,216

For the generators without data (97.6 GW), there's approximately:

25 GW open-loop gas-fired combustion turbines (19% of total capacity for this type)
15 GW gas-fired combustion turbines as part of a combined-cycle plant (10% of total)
10 GW steam coal (4.4% of total)
9.8 GW gas-fired steam turbine as part of a combined-cycle plant (9.5% of total)
6.6 GW wood or wood-waste biomass steam turbine (79% of total)
5.4 GW gas-fired steam turbine (standalone) (7.7% of total)
5.1 GW petroleum-liquids-fired gas turbine (38% of total)
4.8 GW gas-fired internal-combustion (100% of total)
4.6 GW petroleum-liquids-fired internal-combustion (100% of total)
2.1 GW gas-fired single-shaft combined-cycle (16% of total)
2.0 GW municipal solid waste steam turbines (98% of total)
1.4 GW landfill gas internal combustion (100% of total)
5.3 GW other

Assuming we can get good heat rate curves for a majority of the plants with CAMD data, I think we can fill in the missing data reasonably well using the data for that type, e.g. with weighted-average heat-rates or with capacity-to-heat-rate regressions. The classes of generators with mostly missing data seem to fall into either biomass, municipal solid waste, landfill gas, or rarely used backup generators (my assumption for petroleum-liquids-fired gas turbine, gas-fired internal-combustion, and petroleum-liquids-fired internal-combustion). We can probably come up with some decent assumptions for these in another way, and they're probably not especially important to dispatch or capacity decisions.

One more piece of information that we'll need to transform heat rate curves to cost curves: fuel prices. One approach would be to add per-zone fuel prices to const.py.

Closed via #225. Extra features may be added later.


[SPARK-34059][SQL][CORE] Use for/foreach rather than map to make sure execute it eagerly

What changes were proposed in this pull request?
This PR is basically a followup of https://github.com/apache/spark/pull/14332.
Calling map alone might leave it not executed due to lazy evaluation, e.g.)
scala> val foo = Seq(1,2,3)
foo: Seq[Int] = List(1, 2, 3)

scala> foo.map(println)
1
2
3
res0: Seq[Unit] = List((), (), ())

scala> foo.view.map(println)
res1: scala.collection.SeqView[Unit,Seq[_]] = SeqViewM(...)

scala> foo.view.foreach(println)
1
2
3

We should better use foreach to make sure it's executed where the output is unused or Unit.
Why are the changes needed?
To prevent the potential issues by not executing map.
Does this PR introduce any user-facing change?
No, the current codes look not causing any problem for now.
How was this patch tested?
I found these item by running IntelliJ inspection, double checked one by one, and fixed them. These should be all instances across the codebase ideally.

cc @srowen FYI

cc @srowen FYI

Test build #133886 has started for PR 31110 at commit f3e864e.

Test build #133886 has started for PR 31110 at commit f3e864e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38475/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38475/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38475/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38475/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38475/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38475/

Test build #133886 has finished for PR 31110 at commit f3e864e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133886 has finished for PR 31110 at commit f3e864e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133886/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133886/

Test build #133890 has started for PR 31110 at commit 74bc183.

Test build #133890 has started for PR 31110 at commit 74bc183.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38479/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38479/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38479/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38479/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38479/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38479/

Test build #133890 has finished for PR 31110 at commit 74bc183.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133890 has finished for PR 31110 at commit 74bc183.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133890/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133890/

Thanks! Merging to master.

Thanks! Merging to master.

Thanks @srowen and @viirya!

Thanks @srowen and @viirya!

Thanks for fixing this @HyukjinKwon ! Surprised these did not cause problems earlier.
Do we want to backport this to older branches as well ?

Thanks for fixing this @HyukjinKwon ! Surprised these did not cause problems earlier.
Do we want to backport this to older branches as well ?

Sure I'll prepare backport

Sure I'll prepare backport

It will definitely have the same effect if the collection is not lazy, so I doubt it currently is the cause of a bug. But yeah it'd be fine to backport in case there is any chance the collection that something returns suddenly gets lazy.

It will definitely have the same effect if the collection is not lazy, so I doubt it currently is the cause of a bug. But yeah it'd be fine to backport in case there is any chance the collection that something returns suddenly gets lazy.


Installation Instructions

At first glance, this package looks like it should be intallable with elm package install powet/elm-pouchdb but that failed (presumably because the package isn't pure elm?).
Just tried npm install powet/elm-pouchdb instead, and that seems to work, but it sure would have been nice to have this in the READMe. ;)
(And now that I'm trying to use the code that the above installed, it seems like my intuitions are still wrong about how to use this. Coming from a Swift/iOS background, so please forgive me if any of this seems naive.)

Closing, as I haven't used elm for awhile and this issue never got traction. :)


OpenCL interoperability does not work on 64-bit Windows

Currently the OpenCL interoperability does not work on 64-bit Windows machine. The reason for this is the use of c_int_t which defaults to long on Windows (which on the other hand is 32-bit). This can be problematic with things like device IDs as they can easily go over 2^32/2-1. For example https://github.com/arrayfire/arrayfire-python/blob/master/arrayfire/opencl.py#L88 uses c_int_t, as well as a few other functions in the same file. This can be fixed by replacing the c_int_t with, for example, c_void_ptr_t as it is used with the context. Is there by the way some specific reason why context is treated differently?
There is also another issue. When the context is created by PyOpenCL, the input device, context and queue values are input as python ints in order to use the same context and queue, but on Windows you get <class 'OverflowError'>: int too long to convert errors then. These can be avoided by casting them to, for example, c_int64-types. For example in https://github.com/arrayfire/arrayfire-python/blob/master/arrayfire/opencl.py#L156, replace safe_call(backend.get().afcl_add_device_context(dev, ctx, que)) with safe_call(backend.get().afcl_add_device_context(ct.c_int64(dev), ct.c_int64(ctx), ct.c_int64(que))) (using c_void_p seems to work too).
I haven't tested these changes extensively, but they do seem to get things working on 64-bit Windows. I can provide a PR if needed.

Note that the PyOpenCL parts have been commented in the tests: https://github.com/arrayfire/arrayfire-python/blob/master/tests/simple/interop.py#L78, although the issue that is mentioned there has been closed over six years ago.


Adding bindings for predictOne and related methods and attributes

This PR adds the very useful predictOne method binding, at the cost of having to add some adapted methods in the core library.
The use of functions receiving references as parameters seem not approachable, at least for me, so I added mirror functions without them.
I'm not positive about having this PR merged, but it works, and probably will spark the light to a much better approach.

wow. Super cool! Thanks..

I'm not sure If I can sign the CLA. Would you take inspiration in my changes, and add new wrappers from your repo? Thanks!


Datadog metrics are missing relational information

Environment

Telemetry Streaming Version: 1.23
BIG-IP Version: 14.1.4

Summary
Cannot create a relation between pool member and its pool.
We use the Datadog consumer (I believe this issue is not consumer-specific).
In the consumer trace, I see
                   {
                        "host": "ip-10-100-2-55.eu-west-1.compute.internal",
                        "interval": 60,
                        "metric": "f5_test_another.pools.members.serverside.bitsOut",
                        "points": [
                            [
<PHONE_NUMBER>.325,
                                0
                            ]
                        ],
                        "tags": [
                            "addr:<IP_ADDRESS>",
                            "monitorStatus:up",
                            "port:443",
                            "availabilityState:available",
                            "enabledState:enabled",
                            "status.statusReason:Pool member is available"
                        ],
                        "type": "gauge"
                    },

We need to link this information with the pool. A tag such as pool_name is missing.

@dstokesf5 f5_test_another is a metric prefix I'm adding to all metrics (code customization in the meantime https://github.com/F5Networks/f5-telemetry-streaming/issues/158 gets eventually implemented).
The pool name is not showing. As far as I understand, it's (poolName) excluded from the normalization (https://github.com/F5Networks/f5-telemetry-streaming/blob/e9774dbd6118323c7215bcb762f2878451402a9d/src/lib/properties.json#L805-L824)

Thank you for your feedback. I have added this issue to our internal product backlog as AUTOTOOL-2800.

I guess it had been fixed in v1.25.0?

This has been resolved in the 1.25.0 release.


is this with timeline editor in plan?

thanks for the good job

Oh, maybe this can be scheduled, good advice


How to prevent keys collision while using Memchached driver?

I've failed to find the answer in the documentation.
As we know, Memcached has common key namespace for each client. This feature causes keys collisions when multiple web applications use the same Memcached server.
Do I have to perform any explicit actions to prevent collisions? Is the \Stash\Pool::getItem method abstracted from this problem?

When setting up the Memcached driver you can pass options in to the underlying system. The OPT_PREFIX_KEY option will let you specify a key that will essentially namespace things.

@tedivm Does Pool::setNamespace do the same thing (namespaces things)? Which one if preferable: OPT_PREFIX_KEY  or Pool::setNamespace?
As I understand, Stash doesn't separate namespaces for each application allowing them to have common cache. Separating namespaces is option for developers. Do I understand correctly?

The built in namespaces are meant to segment different areas of the same application from each other, but not to separate different applications on the same backend. It won't work that way, as the driver also stores meta data that isn't going to be uniquely namespaced.
I think the OPT_PREFIX_KEY would make the most sense. A unique prefix key for every application will prevent any mingling of data.

@tedivm I got it. Thank you!

No problem, I'm happy to help!


Colors to differentiate days

https://www.reddit.com/r/discordapp/comments/8lev3t/discord_colored_text_with_code_markup_guide/

unfortunately, it only allows for 5 different colours + default and none of the colours work on mobile
and the syntax for events absolutely breaks the formatting

Solarized Red (diff)

requires dash preceding line

Solarized Orange (glsl)

does not highlight properly

Solarized Yellow (fix)

works properly

Solarized blue (md)

does not render

Solarized Cyan (yaml)

dash is red

Solarized Green (CSS)

text before : is exempt from formatting, | and & are not rendered, text after : is rendered orange

Quote ("Brainfuck")

dashes display blue
difficult to see on dark mode and in codeblock

This leaves us with Solarized Yellow and default as viable colours

another reference confirms these problems
https://www.writebots.com/discord-text-formatting/
the only colour we can reliably create is solarized yellow and default, going to close issue as there is no point


Shader system improvements (redo)

An update of #2577 which removes extraneous changes and updates the target branch to 9.4.0-Dev.
Provides the following changes:

Store the original values for the fragment/vertex header such that they can be accessed by child classes.
Store the original values for the fragment/vertex body such that they can be accessed by child classes.
Store the original values for the fragment/vertex source such that they can be accessed by child classes.
Added a configurable glVersion annotation (and corresponding variables) which can be used to modify the GLSL version directive. The default value has been set to 120, which should resolve an issue on some platforms where the uniform type is unavailable.
Added checks to __processGLData to ensure a variable exists before it is assigned via Reflection.

This resolves an issue caused by custom implementation of variables in child classes.
Added caching to the field check to reduce calls to Reflect.fields, improving performance.

These changes were performed to support fixes for FlxRuntimeShader, as part of pull requests HaxeFlixel/flixel-addons#368 and HaxeFlixel/flixel-addons#399. Making changes here prevents code duplication in FlxRuntimeShader, improving maintainability.
These changes have been tested with Friday Night Funkin', where it is used on Windows, MacOS, Linux, and HTML5 platforms.

@Geokureli Let me know if you have any other syntax changes to make, I implemented the ones you suggested.


BCUninstaller crashes when selecting Tools -> Clean up Program Files folder

On two different windows installation (7 and 10), BCUninstaller crashes when when selecting Tools -> Clean up Program Files folder.
Error message:
System.NullReferenceException
System.Object b__28_3(System.Object) @ BulkCrapUninstaller.Forms.JunkRemoveWindow+<>c

Confirmed, will fix.


customize which libraries are loaded and which modules are opened by default

Although one can always include the desired directives and module open commands in the input file, it might be convenient to pass these as command line options. For example, someone might want to always load core and async and open Core.Std and Async.Std, without having to write this at the top of their files.

I believe this is done in 90a3809fdeb9fb0cf1becbcc8e029eff84b07874
I also added a .merlin file in a commit prior to that one.

Thanks. I merged to master. I'll leave this issue open however, as I think we can get a more general solution too. We should be able to customize for libraries beyond just Core and Async.

Do you always want to load core?  It is rather heavy (takes a noticeable time at startup) and is not necessarily needed (considering Oloop is a general library). @agarwal ?

@Chris00 Now that oloop is much more general, no particular library should be given any special consideration. You can load Core just like anything else.


Allianz Tiriac - 404 error when "Vizualizeaza website" button is used.

ENVIRONMENT: production
URL: https://admin.peviitor.ro/
Browser: Chrome
Device: PC
OS: Windows 10
STEPS TO REPRODUCE:
1 .Open the URL in the browser and connect to user.
2. On the page "Companii" type in the searching bar - Allianz-Tiriac and enter
3. In the box of Allianz-Tiriac company click on the button "Vizualizeaza website"
EXPECTED RESULTS:
We are redirected on the website of the company.
ACTUAL RESULTS:
We are redirected on a 404 page with the title " Ne pare rau, insa aceasta pagina nu este disponibila".

Fixed...

retested..PASS..well done !


Allow edit of old purchases under Inventory

Purchase entries once entered are non-editable. Allow editing in a case if there is a mistake.

@aniljava this is intentional.  The idea is that any adjustments to inventory, including mistakes made entering a purchase would be adjusted via the Adjust button on a location.  The reason for this design is that it ensures that any adjustments to inventory have a clear audit trail.

Agreed. Thanks for the input @aniljava. If you have further concerns re: this issue, feel free to reply. As @jkleinsc pointed out, the process for adjustments vs. edits came directly from user feedback.


Unwrapping Applicability for Complex 3D Models

Hi, thanks for sharing your amazing work @QhelDIV
I have a question regarding the UV unwrapping for 3D models.
Is UV unwrapping generally applicable to 3D models that consist of multiple assembled meshes or objects with numerous geometric modifications like holes?

We assume the 3D model is already unwrapped. Typically, to unwrap the complex model, 3D artists first cut it to simpler patches, then perform UV unwrapping on those patches indivisually, which is usually done by surface parameterization algorithms.
So yes, UV unwrapping is general applicable to all kinds of 3D models.


Fix scope removing on modal close

Fix for issue #50. Now modal close doesn't destroy passed scope.

Coverage remained the same at 100.0% when pulling 129c11d1def7054806c925ed7864c14feed9dd64 on rumkin:patch-2 into 30906b394bdfc937f9148700618dee75f2cfc6f3 on dwmkerr:master.

No longer valid as original PR for scopes provided rather than created has been reverted


lexgen: incorporate new tools.ozone namespace; bunch of old lexicons deprecated

DO NOT MERGE YET
This is based off the atproto PR: https://github.com/bluesky-social/atproto/pull/2288
Does mostly expected things, adding api/ozone for tools.ozone.* namespace. Easier than expected because no records in that namespace, so no cborgen needed.
There are some cborgen changes rolled up in this PR. I think I didn't commit changes to at least the bsky codegen, something about string and pointer-to-string. I assume those are unrelated to the namespace stuff and should be resolved in a separate clean PR.
~I haven't self-reviewed or tested this at all, should do that before merging or deploying. I think I caught all the string constants with com.atproto.admin.defs#whatever in code, but there might be more lurking (these don't result in compile-time errors).~

@bnewbold the pointer and string changes are manual fixes that we need for now while a cborgen bug gets worked out. Iirc this has to do with pointers to strings and there's only one spot in the generated code that causes this error. If you manually replace the string(<var>) with &<var> in that line you can get it to build.


Adds Vue Formulate to relevant lists

We're proud to have launched Vue Formulate 2.0 as an open-source form authoring solution for Vue.js. This PR adds Vue Formulate to 3 lists which all apply to the capabilities of the tool.
Additions have been made to the bottom of the lists on:

Generator
Form
Form -> Validation

@Scrum Hello! Definitely still relevant. I've updated to resolve conflicts.

@andrew-boyd ping ?

@Scrum I added Vue Formulate in three places because it applies to each of those three lists. Vue Formulate is a form authoring tool, a form validation tool, and a form generating tool all-in-one. Is it not kosher to have it listed within each section that It applies to?

@andrew-boyd No, I think that the tool has a mainstream and everything else is its options

Ok, I’ll need to re-look at the readme and see where it best applies if there only one entry allowed. I’ll update the PR today, thanks!


More information

Additional information blocks:

Personal characteristics

Attentive to the details
Open to critic
Responsible
Good teammate

Skills

HTML, CSS, JavaScript, tailwind, next.js, trpc, express

Hobbies

Insects & fish
Sport
Travels

Good in

Experience of working in a high stress environment
Communication

Weaknesses

need time to involve into technologies
perfectionism, which sometimes slows down the completion of tasks


[HW] Shrujan Sonawane

Name: Shrujan Sonawane
Email<EMAIL_ADDRESS>Linkedin Profile: https://www.linkedin.com/in/shrujan-sonawane-94bab8217/
Attach the homework step I screenshots below:

Attach the homework step II screenshots below:

Thanks Shrujan! FYI I had to remove an image as it contained your astra token, I would recommend deleting the token for security.

Hey Shrujan Sonawane, great job! Congrats! Here is your badge! https://api.badgr.io/public/assertions/TaLMTnPEQ6-_a577MPFd0Q


ElementNotVisibleError is deprecated

While am using the wait conditions in the pageobject am getting the following error. Can any one resolve this issue.
19-08-16 08:23:42 WARN Selenium [DEPRECATION] Selenium::WebDriver::Error::ElementNotVisibleError is deprecated. Use Selenium::WebDriver::Error::ElementNotInteractableError (ensure the driver supports W3C WebDriver specification) instead.

@anilreddy this will be a problem in Watir. There is a pull request to Watir (https://github.com/watir/watir/pull/867) to address this.

@jkotests I am getting these warnings as well it is in pageobject.
2019-09-23 01:27:54 WARN Watir ["visible_element"] #visible? behavior will be changing slightly, consider switching to #present? (more details: http://watir.com/element-existentialism/)

This one as well.
2019-09-23 01:42:51 WARN Watir [DEPRECATION] ["stale_present"] Checking `#present? == false` to determine a stale element is deprecated. Use `#stale? == true` instead; see explanation for this deprecation: http://watir.com/staleness-changes

@anilreddy , can you please provide code examples for when the error is being generated? It would help us track down where the problem is. Thanks.

Am trying to find similar example which I got in the project. I will post it once I find it.


TS-5104: Correct the calculation of max_dns_lookups in OSDNSLookup

The url expansion feature has removed from 7.0.x. only fix it in 6.2.x branch.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/1276/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1171/ for details.

[approve ci]

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1270/ for details.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/1377/ for details.


[Docs] Vcpkg via FetchContent

Show how to easily incorporate vcpkg into a CMake project without git submodules

@microsoft-github-policy-service agree

Please add - [Vcpkg via FetchContent](#vcpkg-via-FetchContent) into the line 36.

I changed the cmake min to 3.14 since that's when FetchContent_MakeAvailable was introduced.

I think this PR looks good now based on the feedback


the buffer for access log is too large

Currently we configure the buffer of the access log like this:
    access_log {* http.access_log *} main buffer=32768 flush=3;

According to Nginx's doc:

The buffer size must not exceed the size of an atomic write to a disk file

http://nginx.org/en/docs/http/ngx_http_log_module.html
And for Linux, the default atomic write size is PIPE_BUF (4096), see man 7 pipe.
Therefore, the default size is too large to be safe.

+1
A log buffer that is too small is not good, 4096 is more appropriate.

On the other hand, is flush=1 better?
Since the log buffer is lowered, the flush time should also be lowered.

https://github.com/apache/incubator-apisix/pull/1570


Run tests against java 17

Building with JDK 17 looks good so far. Maven build passes, so no adjustments to jte required 👍 Users out there should be able to use the latest jte version + JDK 17 with no problem.
However, Gradle is not yet ready for JDK 17:
https://github.com/gradle/gradle/issues/16857
This can be merged as soon as Gradle 7.3 is released with official support for 17.

Codecov Report

Merging #115 (e88b3e5) into master (46f4636) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #115   +/-   ##
=========================================
  Coverage     92.40%   92.40%
  Complexity     1092     1092
=========================================
  Files            58       58
  Lines          2778     2778
  Branches        443      443
=========================================
  Hits           2567     2567
  Misses          111      111
  Partials        100      100

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46f4636...e88b3e5. Read the comment docs.


Ontology graphs

(migrated from the BioJS wishlist)
e.g. OWL, OBO

(from the BioJS workshop in June 2012)
Graph view of ontology terms. Proposed by @muthuvenka http://www.ebi.ac.uk/chebi/chebiOntology.do?chebiId=CHEBI:35135

Just stumbled across this issue and I want to add this to the 'list'. Are issues with the wishlist tag actually (ever) closed ?


refactor: improve init experience

Summary:
Briefly describe the changes made in this PR.
Issue Reference(s):
Fixes #698... (Replace "..." with the issue number)
Build & Testing:

[x] I ran cargo test successfully.
[x] I have run ./lint.sh --mode=fix to fix all linting issues raised by ./lint.sh --mode=check.

Checklist:

[ ] I have added relevant unit & integration tests.
[ ] I have updated the documentation website accordingly.
[x] I have performed a self-review of my own code.

Did not saw @neo773 was still working on it.
Feel free to use the code written here, I'm closing this one @neo773


docs: update readmes

Description

Adds information about the submodule to the main readme file.
Makes editorial updates to the example readme files.

Addresses https://github.com/terraform-ibm-modules/terraform-ibm-transit-gateway/issues/262
Types of changes in this PR
Changes that affect the core Terraform module or submodules

[ ] Bug fix
[ ] New feature
[ ] Dependency update

Changes that don't affect the core Terraform module or submodules

[ ] Examples or tests (addition or updates of examples or tests)
[x] Documentation update
[ ] CI-related update (pipeline, etc.)
[ ] Other

Release required?
Identify the type of release. For information about the changes in a semantic versioning release, see Release versioning.

[x] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content
If a release is required, replace this text with information that users need to know about the release. Write the release notes to help users understand the changes, and include information about how to update from the previous version.
Your notes help the merger write the commit message for the PR that is published in the release notes for the module.
Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

Merge actions for mergers

Use a relevant conventional commit message that is based on the PR contents and any release notes provided by the PR author. The commit message determines whether a new version of the module is needed, and if so, which semver increment to use (major, minor, or patch).
Merge by using "Squash and merge".

/run pipeline

:tada: This PR is included in version 2.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


proposal: cmd/link: add option to enable full RELRO for ELF

Motivation
Go binaries can be built as position independent executables (PIE) for linux targets which enables partial read-only relocation (RELRO) by default. Partial RELRO as the name suggests does not provide full protection from .got overwrite attacks. When using the external linker it is possible to pass specific linker flags including -z,now to enable full RELRO. Go internal linker, however, does not have the option to enable full RELRO for resulting binaries.
Also, go's internal linker has three notable differences from the other known ELF linkers (ld, lld, bfd, gold and mold) when building binaries with -buildmode=pie and partial RELRO implied:

.dynamic section is not part of the GNU_RELRO segment;
.got section is not part of the GNU_RELRO segment;
GNU_RELRO segment does not have the read-only flag.

Proposal
Add the -bindnow linker option to enable full RELRO on ELF targets that covers both internal and external linkers.
This option defaults to false and preserves current behavior - partial RELRO for -buildmode=pie.
Also, the following changes are proposed to align internal linker's behavior with external ELF linkers:

GNU_RELRO segment is marked Read-only;
.dynamic is a relro section for partial and full RELRO;
.got is a relro section for partial and full RELRO;
.got.plt is a relro section for full RELRO only.

Relevant active CL/PR

https://go-review.googlesource.com/c/go/+/473495
https://github.com/golang/go/pull/58869

Closing since golang.org/cl/473495 has been merged.


Feat: Add ApiKey middleware authentication

Close #108

We need to authenticate our client to our server. Right now it gets 401 but this will be fixed in #82.

We need to authenticate our client to our server. Right now it gets 401 but this will be fixed in #82.

In that case I would argue we need both of those issues in a single PR. The development branch must be working correctly.

Okay, I'll fix it. I thought we wanted two different issues for this.


Cannot open photo library

on ios,
when I click the upload image button
it shows only camera, no "photo library" option

just remove :capture="capture" from the component

Not tested: Maybe you can try setting :capture="false" in view instead of remove the code.

Thanks @adapterdigital I'll have false set as the default value in the next update.

Resolved in 2.1.1


Errors are not shown in stand alone files

Describe the bug**
With the latest version of VS and Ionide, errors are not shown in a script file.
Steps to reproduce

Open VS Code (used version etc.. see below).
Choose new file and set the language to F#
Write some code with obvious errors

Link to sample reproduction
let xs = [ "a"; 1 ]

Expected behaviour
There should be an error sign under the above code
Screenshots

However, no error is shown.
Machine info
Version: 1.54.2 (system setup)
Commit: fd6f3bce6709b121a895d042d343d71f317d74e7
Date: 2021-03-11T00:56:19.848Z
Electron: 11.3.0
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 6.1.7601
Ionide-fsharp v5.4.0

Important to note here that you're in an un-saved script file. I can't repro this in a saved script file. Our processing of untitled documents in FSAC could have regressed.

@baronfel You're right upon save the error displays. Only I have another script file that won't show the errors.

But why in this script file and not the other??
One thing to note, this script file is 1000 lines. So, could that be the problem?

I can't tell anything really without the full contents of the file. Do any errors at all appear? Is it only this kind of error that doesn't appear, etc, etc.

@baronfel No errors appear at all, indeed. I know this isn't really a clear reproducible problem. However, when I just delete all proceeding code, still no errors appear:

@halcwb , try

Close the file
Close the folder using File > Close folder dialog
Reopen folder using the File > open folder (or open recent folder) dialog
reopen script file.

This tends to work for me. If this fixes it, then it's probably a similar bug to https://github.com/ionide/ionide-vscode-fsharp/issues/1500

@nhirschey Difficult to say. Unfortunately, the problem is all over the place and seemingly inconsistent. For example, I tried your method without success and as shown below, the error is detected in the file view, but doesn't show in the editor window:

That screenshot is of ErrorBug.fs. Is ErrorBug.fsx working now in editor view after following my steps?

@nhirschey Oops, you're right. With small simple script files your solution works. However, with a large script file (including a typeprovider), the error messages never show up, whatever I do.

The frustrating thing is that currently I cannot run the script file using any editor. Visual Studio can't handle net5 libs. Rider, the same and VSCode isn't giving any error warnings (also is quite slow). VS for Mac can't handle script files at all. I tried them all.

Visual Studio supports running a net5 script, and has full IntelliSense for it in the 16.10 previews. You need to turn it on in the settings though (it will be the default in the next major version of VS). Can't speak towards Ionide or Rider at the moment.

@baronfel Your last bullet on the list is the issue at hand. I am struggling to get a repro. It's a long script file using a type provider. I can send you the code file? The strange thing is if I delete all the code in the script file and do a save as, still no errors show in that script file when you type some invalid code.
@cartermp Can I download the preview? Because with the current version with use of net core scripting enabled, it doesn't work.

I guess some clarity is in order: when you say 'run', do you actually mean run? like using the send to FSI command in VS code and FSI running the script? or are you meaning that typechecking/editor features don't work for that script? it sounds like the latter, but the phrasing is vague enough that I want to pin down what you mean.

@baronfel It's specifically the error checking. I can indeed run the script. But when I start editing the script I get the intellisense, the code completion but no error detection. I think the screenshots are quite clear.

@halcwb , see here for Phillip's post about .net core scripting in preview Visual studio and how to enable it: https://devblogs.microsoft.com/dotnet/f-and-f-tools-update-for-visual-studio-16-9/
The scripts of mine that had trouble in ionide were properly type-checked if I opened them in VS2019 instead, so .net core scripting in VS19 should work. Though now I just follow the close/reopen procedure to get ionide type-checking back whenever it happens. It's not a one-time fix; you may have to do it a couple times on the same file over the span of a couple hours.

@halcwb can you add the schema to that gist as well, or a description of it?

@baronfel , I don’t know how the tooling works, so this is a long shot but my scripts where typechecking randomly stops working in ionide also reference FSharp.Stats and Plotly.NET. I can’t imagine that matters, but it’s a similarity so I thought I’d mention it.

@nhirschey for a brief explainer, in order to typecheck we need to pieces of data:

the contents of the file to typecheck, and
a set of options describing what parameters to use to configure the compiler when checking the file (this includes references, #defines, target framework, etc)

The compiler services have helpers that help generate the options for scripts, and the options for files-in-a-project are derived from the project file.
It could be that in the code we have to handle scripts there's some condition that results in the options getting modified or corrupted, but there's not any good monitoring or operational logging around that at the moment to help us debug.

@baronfel I have attached the two schema files:
shema files.zip
A couple of things to note:

Due to work restrictions I used VSCode on a Windows 7 PC. But now I also tried a Windows 10 PC installation of VSCode. And then the bug behaves exactly as described by @nhirschey, with the same work around.
Also using Windows 10, VSCode seems a lot faster (PC specs of the Win10 are actually way below the specs of the Win7 PC).

See with that I will close this bug as a duplicate of https://github.com/ionide/ionide-vscode-fsharp/issues/1500.
Thanks for listening in and the help @baronfel @nhirschey @cartermp. Really appreciate it.


Is there an option for specifying a subdirectory to watch?

Hi there,
I have a 'deploy' directory where all of my gulp tasks output to. Is it possible for me to store the theme config.yml file in my top level directory, and have it reference the deploy directory?
That way tasks can be run from the top level directory without having to change to the subdirectory to run theme tasks.
Is there a way to do this?
Thank you very much :)
Adam

Hey @adamwooding can you give me an example of what your directory structure looks like? Could't you use something like foreman to be running the gulp scripts in the top directory and run your theme task in your child directory in parallel?


Fix Slider a11y properties

We currently have ariaLabelledby property which is mandatory on Slider. Not only does it differ from native prop name, but there are other valid combinations using aria attrs which are valid for Slider.
https://www.w3.org/TR/wai-aria-practices/#slider_roles_states_props
Actions

Make ariaLabelledby optional and make note of removing it in future releases as native aria-labelledby should be used.
Are other a11y combinations with aria-value/min/max/now upheld?
Accessible aria-valuetext story?

The aria-labelledby does not seem to work on range-slider (aria-labelledby is added to the wrapper div which has a role of group on the range slider and directly on the input on the simple slider). Tested with mac voiceover and windows narrator.
The {...rest} is added to the wrapper-div in both cases so this will probably need to be changed to the inputs if we want other aria stuff to work.
The ariaLabelledby prop value is used to create ids for the input(s) (which is used for internal sreen-reader only labels) but maybe we can use useId hook for this


As a user, I can list vaccination

... to clear my previous filters that I have made on vaccination

Closed in #230


Improve logic for checking when to proceed to next fix

Especially on a slow computer or at higher simulation rates, it is noticeable that aircraft miss their waypoints slightly, because the distance covered between two position recalculations is outside the radius acceptable to consider the aircraft to be "over the fix". Instead, it now checks whether they have PASSED the fix, thus resolving the issue.
This pull resolves #456.
Demo: erikquinn.github.io/atc/b/loop-dee-loop
Before-After Comparison: https://youtu.be/EahsbmySFPQ

I think this has a bug, if you order an aircraft to a fix which would require a greater than 180 degree turn then it will immediately proceed to the next fix, even if the requested fix is 100km away.  I'm not sure the best way of fixing that, perhaps keeping the previous offset and switching to the next fix when the offset goes from positive to negative?

@tedrek Yeah, you're right. I thought about this for a long while today, and can't come up with a good solution, sadly... Best I can seem to do is to limit it so it will only move on to the next waypoint if it has passed it and is within a certain distance of the fix. Then, worst case scenario, it would skip the waypoint that's close behind them, and then enter the turn to the second one... it's not ideal, but would at least solve the main issue at hand I guess.

An unstated assumption here is that the game should behave well as the simulation step size increases without bound.  However that's probably a bad approach to take.
assets/scripts/module.js:304 limits the step size to 1/20 of a second.  With 5x warp the step size can increase to 1/4 of a second.  My thinking at this point is the step size should be limited to 1/5 of a second as the value at which two aircraft will reliably collide when in a head-on course at speed and altitude.  This means an aircraft would not move more than roughly 50 meters in a single step.
#456 is not caused by a slow computer, rather it's caused by the step time being increased to 5 seconds during projection.  This means aircraft can be moving up to 1.25 km per step which makes it pretty clear why the 1km distance doesn't work reliably for projections.

@tedrek okay, understood.  I fling the simulation rate up to about 1000x and in less than 5 minutes real time, it's pretty well guaranteed to have spawned all combinations of aircraft, and then I know there aren't any problems with airlines, fleets, SIDs (assuming I depart a few), STARs, etc. It's caught more than a few things I missed when reviewing others' work, and the spinning kind of just drove me bonkers, that's all.
It actually boils down to two things when at high sim rates:

The projection lines aren't always accurate, because the step size can become enormous.
The aircraft might not realize when they "pass" a fix because they hopped over the point.

Paths forward that come to mind:

set a cap of ~300m maximum projection step size
- :+1: resolves item 1
- :x: limits the allowable maximum simulation rate
use changed logic for passing fixes, as proposed in this pr
- :+1: resolves item 1
- :+1: resolves item 2
- :+1: does not limit allowable maximum simulation rate
- :x: when <~5nm from the current fix, facing away, it will skip over that fix (undesirable)
spend fewer cycles on projection and more on positional precision
- :+1: resolves item 1 because more data will be available
- :+1: resolves item 2 because more data will be available
- :+1: end up with a higher density of positional data
- :ok_hand: would have to run some timers that call canvas_draw_aircraft() at a specifyable interval in order to update at a different rate than the position recalculations (but that doesn't worry me)
do not attempt to conduct projections above 50x simulation
- :+1: resolves item 1
- :x: does nothing to address aircraft who actually do miss their fixes (item 2)
- :x: no projection lines, obviously

Way I see it, I'd personally rank the options 3-2-1-4.
How do you feel about these? Or any other ideas you may have?
Thanks.

Option 3 is the plan, if projection isn't just to be removed altogether. Regardless, this can be closed for now.


SetValues gives unexpected results

  public async update(id: string, card: Card): Promise<Card> {
    const results = await this.db
      .matchNode('card', 'Card')
      .where({
        'card.id': id
      })
      .setValues({
        'card.title': card.title,
        'card.type': card.type,
        'card.isActive': card.isActive,
      })
      .return('card')
      .run();
    return CardService.map(results)[0];
  }

produces this error:
Invalid input '+': expected whitespace, comment, '(', '.' or '=' (line 3, column 16 (offset: 53)) "SET card.title += $cardTitle, card.type += $cardType, card.isActive += $cardIsActive"
                ^
Neo4jError: Invalid input '+': expected whitespace, comment, '(', '.' or '=' (line 3, column 16 (offset: 53))
"SET card.title += $cardTitle, card.type += $cardType, card.isActive += $cardIsActive"

I assume this is a bug and not a feature? Ima go ahead and remove that '+' sign and shoot you a pull request! On the other hand Ima not set you a pr. I got no clue where the override is set to true in:
  setValues(values: Dictionary<any>, override?: boolean) {
    return this.chain(this.wrap(new Set({ values }, { override })));
  }

Ive forked it and ill hack the override: false for now so I can continue... but great work btw thank you so much for that!

I can see the problem, I'll investigate it when I get a chance. In the mean time you can fix the issue by calling set values like this:
.setValues({
  'card.title': card.title,
  'card.type': card.type,
  'card.isActive': card.isActive,
}, { override: true })

When override is true, it will use the = operator instead of +=.

:tada: This issue has been resolved in version 3.5.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix issue #197: create lock before lib file, remove lib files by absolute path

We faced this problem running a few instances of app that use sqlite. Creating lock-file before lib allows to avoid concurrent creating/deleting on initializing. Test taken from @bertmaher 's repo, reproduces on MacOS 10.12.5. Extra tested on Win10

Merged. Thanks for the fix.


Batch size count issue

Hi,
With increase of batch size count to 20 and batch size bytes to 20Kb, i see the events are dropping in Splunk. Anyone faced this issue earlier?
Thanks
Teja

@tejadivvi I am working with similar situations.
Your message contains way too little information for people to reply. What do you mean by dropping - that you lose events in the flight?
Please share more if the issue is still valid, or ask it to be closed if it's not. Thank you.

Hi @tejadivvi,
Since there is no updates for this issue. Assuming it as resolved for now.
Please let us know, if you still face any issue.
Closing this issue.


downgrade pygments requirement to 2.6.1

Unless I'm mistaken, we don't actually need 2.7.1 for this, and downgrading the requirement to 2.6.1 makes it possible to get TestSlide packaged in Fedora 33.

That's reasonable. Thanks for carrying on the Fedora packaging :-D


Implement a ThreadLocalCache to improve read perf

Change-Id: I372fb5a56f374fc3993d8bdcbbb0ef07b5f160c7

After the loader performance test, there is no obvious performance improvement for the two caches, so close this pr