feat: rename TopLevelObject

Drop all short hands and clever names for explict types

Codecov Report

Merging #117 into master will not change coverage.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master     #117   +/-   ##
=========================================
  Coverage     90.07%   90.07%
  Complexity      155      155
=========================================
  Files            47       47
  Lines           504      504
  Branches        116      116
=========================================
  Hits            454      454
  Misses           10       10
  Partials         40       40

Impacted Files
Coverage Δ
Complexity Δ

src/main/kotlin/com/expedia/graphql/toSchema.kt
100% <ø> (ø)
0 <0> (ø)
:arrow_down:

.../main/kotlin/com/expedia/graphql/TopLevelObject.kt
100% <100%> (ø)
1 <1> (?)

...n/com/expedia/graphql/generator/SchemaGenerator.kt
100% <100%> (ø)
27 <0> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a1b189...a15157d. Read the comment docs.


gym export_options : {method} resolves into empty string

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description
I am using build_app function to build an iOS app. The execution fails due to the following error:
There seems to be a mismatch between your provided `export_method` in gym
and the selected provisioning profiles. You passed the following options:
export_method:

The export method seems to be resolved as an empty string or a " " character.
The weird think is that "Summary for gym 2.219.0" shows a correct input with all values completed as expected. For disclosure I tried executing with method being "ad-hoc" or "app-store".
Is there any reason for gym translating the input to an empty string?
Command executed
disable_automatic_code_signing(
            team_id: options[:team_id],
            profile_name: options[:provisioning_profile],
            code_sign_identity: options[:code_sign_identity]
        )

  increment_build_number(
            build_number: ENV["BUILD_NUMBER"]
        )

build_app(
            derived_data_path: options[:derived_data_path],
            workspace: options[:workspace],
            scheme: options[:scheme],
            include_symbols: true,
            toolchain: "com.guardsquare.ixguard",
            output_directory: options[:output_directory],
            output_name: options[:output_name],
            clean: true,
            codesigning_identity:options[:codesigning_identity],
            export_options: {
                teamID: options[:team_id],
                method: 'app-store'
            }
        )

Complete output when running fastlane, including the stack trace and command used
There seems to be a mismatch between your provided `export_method` in gym
[17:39:20]: and the selected provisioning profiles. You passed the following options:
[17:39:20]:   export_method:
[17:39:20]:   Bundle identifier:  com.xxxxxxxx.qa
[17:39:20]:   Profile name:       com.xxxxxxxx.qa AppStore
[17:39:20]:   Profile type:       app-store

Great post. It was much needed. Love your simplistic style of explanation.

Removing this line of code solved my issue.
codesigning_identity:options[:codesigning_identity],
Looks like there was double definition of signing identity which was causing the issue

Removing this line of code solved my issue.
codesigning_identity:options[:codesigning_identity],
Looks like there was double definition of signing identity which was causing the issue


Handle skipped cols Joi conflict

I attempted to squash the WIP commit into the latest one but I'm scared cuz it wasn't working right for me using git rebase -i HEAD~2

Introduce new return code: MIGRATION_WITH_CONFLICT to inform the user that they have a Joi schema prop that conflicts with a skipped, unsupported db column
Introduce schemaConflict property on all skippedCols entries so a user can search for which columns are an issue

Coverage remained the same at 100.0% when pulling 251923195eb1b35808d4d967015f065245e04c5f on handle-skipped-cols-joi-conflict into 1ff6d25f9d8a0f84c1536f568e41d2b228475224 on prelaunch-tidy.

Coverage remained the same at 100.0% when pulling 251923195eb1b35808d4d967015f065245e04c5f on handle-skipped-cols-joi-conflict into 1ff6d25f9d8a0f84c1536f568e41d2b228475224 on prelaunch-tidy.

Coverage remained the same at 100.0% when pulling 251923195eb1b35808d4d967015f065245e04c5f on handle-skipped-cols-joi-conflict into 1ff6d25f9d8a0f84c1536f568e41d2b228475224 on prelaunch-tidy.

Coverage remained the same at 100.0% when pulling 251923195eb1b35808d4d967015f065245e04c5f on handle-skipped-cols-joi-conflict into 1ff6d25f9d8a0f84c1536f568e41d2b228475224 on prelaunch-tidy.

Coverage remained the same at 100.0% when pulling 251923195eb1b35808d4d967015f065245e04c5f on handle-skipped-cols-joi-conflict into 1ff6d25f9d8a0f84c1536f568e41d2b228475224 on prelaunch-tidy.

Coverage remained the same at 100.0% when pulling 251923195eb1b35808d4d967015f065245e04c5f on handle-skipped-cols-joi-conflict into 1ff6d25f9d8a0f84c1536f568e41d2b228475224 on prelaunch-tidy.

I've been looking at this for a little while and don't have lots of specific feedback yet.  But I do have some more general feedback.  I get confused by the two very different paths that join tables and other tables go down.  For example, join tables require skipped columns to be excluded by recreating a joi schema without certain fields which causes all sorts of complications around setting schemaConflict and actually constructing the joi schema.  But at the same time all models go through a process of filtering their "compiler schema" to do the same thing (which seems like a good approach).  This seems to be related to the duplicated join table info, too.  These might be issues—not 100% sure—but overall the theme for me is that i struggle to grasp the big picture!  Maybe you could break it down for me some time.

Ok yeah! I can definitely give rethinking some of the structure a thought, the join tables though are pretty special lil beasts in this -- and I agree the 'flow' of how the internals work do seem somewhat scrambled about

I'd also be interested in a breakdown if it happens.  There's a lot going on here, and I had a hard time getting my head around it.

Ok howbout at some point I can pick a couple situations found in the tests and we can meet IRL to discuss the logic flow that produced it?

@devinivy Is this PR good to merge and then do the opt-in npm tagged release?

NOTE we are going to remove MIGRATION_WITH_CONFLICT as a returnCode in a future version


gh repo clone Hyperclaw79/PokeBall-SelfBot-Poketwo-Autocatcher

Title: [Docs/Bug Fix/Enhancement/New Feature] Some relevant name for the PR
Body:
Summary:
Here, write up a brief description about the PR and its overview.
Changelog:
List out all the changes in one place.
Modification Steps:
Mention the steps required to make relevant changes and the files which will be affected.
Preview.
Show a screenshot of the bot using these changes on our local end.
Ignore for Docs.

Ninjask


Add new actions for IP address valuation to Linux Pack.

Complete!

Add ipaddress module to requirements.txt in linux pack.
Add new actions is_valid_ip, is_valid_ipv4 and is_valid_ipv6
to linux pack.
Add tests for is_valid_ip.py action script.
Add entry to for new actions to the CHANGELOG.rst.

Thanks.
The changes look good to me :+1:

Please also sync this branch with latest master since there are some conflicts it looks like.

Sorry, I seem to have sync'ed our master branch and then branched from the wrong place :-( I'll fix.

Since this package is a port of Python 3 functionality, I believe it should also work on Windows.
In that case, core pack is maybe a better place for this action. What do others think (in any case, it's not such a big deal right now since we don't really officially support running StackStorm on Windows anyway)?
/cc @lakshmi-kannan @manasdk

I've corrected the Action suffix and brought my branch up to date, so should be fine to merge.

Thanks.
I'll wait for the feedback on the pack and merge it after that :)

I like this in linux pack just fine but I have problems with linux pack being part of st2 code. I lost that argument before but I am not giving up.

It appears no one has a problem moving linux pack back to st2contrib. Once this PR gets merged, I'll own moving it to st2contrib.

I am not very clear what these actions have to do anything with linux or with core StackStorm functionality. Would it make more sense to place them in st2contrib in a standalone pack called ip_utils?

They are general utility / helper functions for use in other workflows,so following a quick discussion on Slack it was thought the linux pack in ST2 would be best location.
I think the real question is Where do you put general propose helper actions? They are of use only in other workflows, if they are not the main st2 repo will folks be allowed to use them in other st2contrib workflows? As the normal review comment is often don't depend on other packs if you use action from another pack.
My suggestion is move them to a new utils pack in st2, which purpose is useful helper actions for workflows. We can then add actions like the following:

is_valid_hostname
is_valid_dns_entry
is_in_dns

Which would all be every useful for adding checks within workflows, I guess the other option wold be aiming to push them into Mistral? Which would reduce the load on the action runners..
These came from my need to add a check for a valid IP in one of our troubleshooting flows and I thought it made sense to share with the wider community as it's action I see others will need to do, so why not same them the effort.

I'm personally fine with either moving it to a separate pack (e.g. networking / ip_utils) or leaving it in a linux pack.
We already have some "similar" functionality in Linux pack (e.g. wait_for_ssh) so I don't see it as a big problem.

I like this in linux pack just fine but I have problems with linux pack being part of st2 code. I lost that argument before but I am not giving up.

This also means updating installer script and other stuff to install this pack by default? Or?

We already have some "similar" functionality in Linux pack (e.g. wait_for_ssh) so I don't see it as a big problem.

I guess we all draw our personal lines very differently. What is the similarity between wait_for_ssh and is_valid_ip  etc? The fact that it has something to do with ip addresses or is related to networking?

I guess we all draw our personal lines very differently. What is the similarity between wait_for_ssh and is_valid_ip etc? The fact that it has something to do with ip addresses or is related to networking?

Both.
Anyway, to unblock this PR, is everyone OK with moving this to new "networking" or similar pack in st2contrib?

If you let me know where it would go, I'll open a new PR.

How about networking_utils in st2contrib?

@jjim sounds good! New networking_utils it is

I've moved this to StackStorm/st2contrib/#498.

I've moved these actions to the networking-utils pack in StackStorm/st2contrib#498.


Conflicting API docs for RenderObjects

The following screenshot shows the current API docs for the detach() method in AbstractNode. The instructions appear to conflict. We're told "first call their inherited detach method, and then detach all their children", but we're also told "this method should end with a call to the inherited method, as in super.detach()". We're told that "super.detach()" should be the first thing called, but also the last thing called.
There may be similar issues with other docs. The line "Implementations of this method should end with a call to the inherited method, as in super.detach()." is repeated in a similar fashion for some other similar APIs. I suggest doing an audit.

Hello @matthew-carroll. Thank you for raising this issue. Labeling for insights and visibility from the team

I did a quick grep for "should end with". There are indeed several methods that have the same contradiction in their docs:

RenderObject.detach (the original report — and also the one I just ran into myself, causing me to find this thread)
Layer.detach
AbstractNode.detach

The first two of those were copied recently from the latter, in the deprecation of AbstractNode:

#128973
#128467

Then there are three methods that have the same "should end with a call to the inherited method" language but are fine, with no contradictions:

State.deactivate
State.dispose
Element.deactivate

I also grepped for "should start with" (and "should begin with", which had no results). That found several methods that say that, and redundantly (but consistently) also have the "first call their inherited … method" language:

RenderObject.attach
Layer.attach
AbstractNode.attach

as well as several methods on State and Element and elsewhere that have the "should start with" without the redundancy.

So the audit comes down to:

The contradiction appears in those three detach methods (which were one method until very recently).
The corresponding three attach methods have a redundancy.

That redundancy on attach is in itself a bit confusing for the reader. Both would be good to fix.
That leaves the question of which way around is the right one for detach. Typically one would expect to detach in the opposite order from attaching, so that'd mean ending with the super.detach() call (since the attach docs are consistent that super.attach() comes first).
Then looking at the Git history to see if that sheds any light… well, that text was added here:

#76021

Before that, there was only one version on each of attach and detach, both saying to call super first. Seems like the author and reviewers of #76021 just didn't notice the existing text there.
And that "first call their inherited … method" text on both methods was added in 2017 in:

#9456

There's no discussion in the docs or in #9456 of why attach and detach surprisingly call for the same order. So my guess would be that the version added later in #76021 is the right one. If it even matters, that is — there's plenty of render objects doing it each way, and it may be that the order doesn't matter in practice anyway.
I guess the next step is to send a PR, and any further discussion of what the right answer is can go in review there.

There's no discussion in the docs or in #9456 of why attach and detach surprisingly call for the same order. So my guess would be that the version added later in #76021 is the right one. If it even matters, that is — there's plenty of render objects doing it each way, and it may be that the order doesn't matter in practice anyway.

Ah, no. On looking closer at the implementation of detach in these base classes, there's an assertion that would fire if a child is detached before its parent calls super.detach. So in fact super.detach must be called before detaching any of the children, just as the older part of the text says.
And looking again at those implementations on various render objects, many of them don't start with super.detach()… but the things they do before that are stopping controllers and removing listeners, not detaching children.
Anyway, I'll reflect those findings in the PR I'll send shortly.


Python: hardcode abs path to library in ffi.dlopen('library') in case of CFFI module

More packages are starting to use cffi in ABI-mode to load libraries. Setting LD_LIBRARY_PATH helps finding the needed libraries and allows a package to be build, installed and tested. However, afterwards the package becomes unusable because the library cannot be found anymore.
So far I used sed to replace the argument to dlopen with an absolute path. Maybe we can do it a bit nicer?

Duplicates https://github.com/NixOS/nixpkgs/issues/7307


method ProtectSheet can lock sheet, but cannot select locked cell

Description

Steps to reproduce the issue:
1.
write a test to reproduce ProtectSheet
func TestProtectSheet(t *testing.T) {
	f := excelize.NewFile()
	name := f.GetSheetName(0)
	err := f.SetCellStr(name, "A1", "you can choose me")
	if err != nil {
		panic(err)
	}
	//
	err = f.ProtectSheet(name, nil)
	if err != nil {
		panic(err)
	}
	f.SaveAs("test_protect.xlsx")
}

Describe the results you received:
without green border, cannot be selected

Describe the results you expected:
with green border, can be selected

Output of go version:
go version go1.14 darwin/amd64

Excelize version or commit ID:
v2.3.1

Environment details (OS, Microsoft Excel™ version, physical, etc.):

this issue is the clone of https://github.com/360EntSecGroup-Skylar/excelize/issues/733

Hi @hybnew, if you wanna make cell selectable, just specify SelectLockedCells as false:
if err := f.ProtectSheet("Sheet1", &excelize.FormatSheetProtection{
	EditScenarios:     false,
	SelectLockedCells: false,
}); err != nil {
	fmt.Println(err)
	return
}

@xuri  it works! thanks buddy! but why SelectLockedCells: false? i thought i would be true.

The false means that NOT protect cell selection, so you can select any cells.


IPS not setting new rules to drop

I enabled 2 new rule sets and changed them to drop. Upon reviewing the rules after in the rule section tab and doing a search for them. Instead of them saying drop it still said alert. The expected behaviour should of been to change them to drop.

@L1ghtn1ng did you "Download & Update Rules" again after changing the set to drop? If so, please inspect the timestamps of the files to check if your download was actually completed (/usr/local/etc/suricata/rules/).
If possible, can you first use the forum (https://forum.opnsense.org/) before opening new issues? This gives others the opportunity to discuss and help, which increases the chance that issues which are logged contain enough information to solve them.

@AdSchellevis No I did not

Closing due to user error. Thanks @AdSchellevis for pointing out what I did wrong


No Links Found/Blocked by Filename

Hi,
I've recently moved from OSMC to OpenELEC on my RPi2, and I am having real issues getting Pulsar to play any TV show. I've run into two distinct errors in the logs that I can see, the most pertinent one being that it appears a string is being appended on to all outgoing search links.
Loglines like this:
 11:47:14 T:1484776512  NOTICE: [script.pulsar.TOTALLYLEGITPROVIDER-mc] https://TOTALLEYLEGITBAY.se/search/SHOW%20NAME%20s01e05%20%23tv%26filter/0/99/200
The issue being the #tv&filter
I can see on the screen, in the notification that pops up for each provider there is the text #tv&filter
http://xbmclogs.com/pyvxohm00
http://xbmclogs.com/p0j4gmcqm
I tried an older version of Pulsar, 0.31, which is what all the providers say "ONLY SUPPORTS", still the same issue.
I'm going to try an older version of ELEC.

From what I'm reading, your provider is filtering out the links by itself, sending nothing to Pulsar

That's what it looks like to me too. I was going through the provider setup documentation, but it's a bit above my level. Thanks for the help. I don't suppose you could direct me to someone who is familiar with the providers? I think that my next step needs to be finding someone who can answer why the providers are all doing this.


docker takes forever to start when /var/lib/docker/volumes is huge

On some nodes, docker takes many 10s of minutes to start.
$ sudo ls /var/lib/docker/volumes | wc -l
16641
$ sudo du -hs
133M	/var/lib/docker/volumes
$ sudo docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64
Doing an strace on docker -d shows that it is sequentially parsing all 16k (or more on most nodes) json files. When I manually inspect these files, they see to be referencing locations that no longer exist... thus they are defunct.
This needs to be made much faster or the files need to be garbage collected somehow. 99.999% of these files don't need to be here.

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

Closing since this is a dup of other reports, and is also no longer relevant since these are no longer traversed since 1.7 (which should help your startup time).
In docker 1.9 we should have a top level docker volumes command, and in Docker 1.8 you can specifically used named volumes (and as such re-use the same volume over and over quite easily).

$ sudo docker info
Containers: 76
Images: 102
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 4.1.3-181
Operating System:
CPUs: 24
Total Memory: 94.38 GiB
Name: my.node.name
ID: FGQ7:ZOJP:WLTA:7VLH:5P44:GKZK:6F6Y:CJ7X:N6VE:OAW7:SC5O:NN3Y
$ uname -a
Linux my.node.name 4.1.3-181.el6siri #1 SMP PREEMPT Wed Jul 22 02:28:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
These are bare-hardware nodes.

Oh, awesome. Thanks.


[BUG] The file direct_url.json is not found

Dear author,
I am currently trying to use the noisebase library for my work, but I encountered an issue with the resolve_data_path() function in the data.py file. https://github.com/balintio/noisebase/blob/0bd8e02ffd4e53ec5b0515d55489ab56b247f0ee/noisebase/data.py#L13
In the function, there is a reference to a direct_url.json file, which is used to determine the location of the dataset. However, I was unable to find this file in the repository.
Could you please provide some additional information about this direct_url.json file? What is its purpose, and where can I find it? I would greatly appreciate if you could clarify this for me.
Additionally, if there is an alternative way to specify the dataset path, could you please share the details? I want to ensure that I can properly configure the library to work with my dataset.
Thank you in advance for your assistance. I look forward to your response.
Best regards.

Pip automatically creates the direct_url.json file when you install the package. It's not part of the repository, as it saves data about your specific install configuration.
This part of resolve_data_path checks whether you installed the package as editable, i.e. pip install -e ./noisebase (probably after cloning the repo) or from PyPI, i.e. pip install noisebase. In the former case, the default data location is in the cloned repo folder; in the latter, it's the current working directory.
It's strange that you're missing the direct_url.json file. Can you provide the commands you used to install the package? Can you also provide the error you get from the Python interpreter?
In the meantime, you can give a data_path parameter to datasets to work around the issue, like so:
from noisebase import Noisebase

data_loader = Noisebase(
   'sampleset_v1', # Dataset name
   {
      'data_path': '/mnt/data/noisebase' # Wherever you store our datasets
   }
)

Scripts like nb-download also take a --data_path argument, which functions similarly.
nb-download --data_path="/mnt/data/noisebase" sampleset_v1

Thank you so much for your helpful response.

I used pip install noisebase to install the package.
The error messages from the Python interpreter:

  File "**********\Desktop\code\graduation_design\nppd\src\train.py", line 60, in main
    trainer.fit(
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 987, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 1031, in _run_stage
    self._run_sanity_check()
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 1060, in _run_sanity_check
    val_loop.run()
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\loops\utilities.py", line 182, in _decorator
    return loop_run(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\loops\evaluation_loop.py", line 110, in run
    self.setup_data()
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\loops\evaluation_loop.py", line 166, in setup_data
    dataloaders = _request_dataloader(source)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\connectors\data_connector.py", line 342, in _request_dataloader
    return data_source.dataloader()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\connectors\data_connector.py", line 309, in dataloader
    return call._call_lightning_datamodule_hook(self.instance.trainer, self.name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\pytorch\trainer\call.py", line 179, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\noisebase\loaders\lightning\training_sample_v1.py", line 19, in val_dataloader
    return PytorchDataloader(get_epoch=lambda: self.trainer.current_epoch, **self.loader_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\lightning\fabric\utilities\data.py", line 324, in wrapper
    init(obj, *args, **kwargs)
  File "**********\anaconda3\envs\nppd\Lib\site-packages\noisebase\loaders\torch\training_sample_v1.py", line 200, in __init__
    ds = TrainingSampleDataset(
         ^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\noisebase\loaders\torch\training_sample_v1.py", line 26, in __init__
    data_path = resolve_data_path(data_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\site-packages\noisebase\data.py", line 13, in resolve_data_path
    dist_info = json.loads(metadata.distribution('noisebase').read_text('direct_url.json'))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**********\anaconda3\envs\nppd\Lib\json\__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType

I would like to try the train.py script in your nppd project, but I’m not sure how to configure
from noisebase import Noisebase

data_loader = Noisebase(
   'sampleset_v1', # Dataset name
   {
      'data_path': '/mnt/data/noisebase' # Wherever you store our datasets
   }
)

within the noisebase.lighting.Trainer class. It seems that this class does not provide an interface for the data path.

NPPD uses Hydra to manage configurations. No need to do this now, but you could put this option in nppd/conf/base.yaml:
...
training_data:
  data_path: /mnt/data/noisebase
  samples: 8
  batch_size: 8
  ...

I think I fixed the issue in e57974ca8e970e19f06d6b6064bde1fef677ef8c. Could you try updating the package (pip install noisebase --upgrade) and see if that works?
Pip actually doesn't create the direct_url.json when installing the package from PyPI. I only tested installing the locally built wheel and missed this. 😅

Dear balintio,
Thank you for your response, it’s been very helpful in resolving the previous issue. I appreciate you taking the time to address my questions.
Additionally, I had a follow-up inquiry. I’ve noticed that the dataset you are using appears to be in a custom file format, such as .0 and .zarray. I was wondering if there is a possibility of using more common data formats, such as HDR, PNG, or other publicly available data storage formats? This would allow me to potentially test your solution with my own dataset.
I’m curious to know if supporting more standard data storage formats is something you’ve considered or if there are any technical limitations that prevent that. I’m interested in exploring the potential of your solution, and having the flexibility to use my own dataset would be very helpful.
Please let me know your thoughts on this. I’m grateful for your assistance and look forward to your response.
Best regards.

No worries, I'm happy to help. The point of publicly releasing a project is to make sure people can use it😉
What you're seeing are Zarr files. It's a very fast hierarchical data format, like HDF5, if you know that one. Initially, we tried storing everything in EXR files, but the speed of our data loaders quickly bottlenecked the training process. Loading hundreds of files for every batch just wasn't a scalable approach.
If you already have everything in individual files, packing them into Zarr files following our format should be easy. We don't have such example scripts in the repository yet, but I can add some and help write the script for your specific case. Just make sure you have all the data listed here in some form.
A large part of this difficulty comes down to using per-sample data. We plan to share per-pixel versions of our datasets in June. They will also use the Zarr format, but they should be even easier to convert between individual files.


3.05 branch build failure

Tesseract Version: 3.05.01
Commit Number: latest code from 3.05 branch
Platform: ubuntu 14.04

Current Behavior:
First reported on forum - https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/tesseract-ocr/g8lTxxYiJ2E/KL07qF_LAQAJ
Build is failing with the following:
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2 -std=c++11  -L/usr/local/lib -o tesseract tesseract-tesseractmain.o libtesseract.la  -lrt -lpthread
libtool: link: g++ -g -O2 -std=c++11 -o .libs/tesseract tesseract-tesseractmain.o  -L/usr/local/lib ./.libs/libtesseract.so -lrt -lpthread
/usr/bin/ld: tesseract-tesseractmain.o: undefined reference to symbol 'lept_free'
//usr/local/lib/liblept.so.5: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [tesseract] Error 1
make[2]: Leaving directory `/home/sanskrit/tesseract-3.05/api'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/sanskrit/tesseract-3.05'
make: *** [all] Error 2

This is using the latest code from github for leptonica as well as tesseract 3.05 branch.
Build was done using following commands:
git clone -q --branch=3.05 https://github.com/tesseract-ocr/tesseract.git tesseract-3.05
cd tesseract-3.05
./autogen.sh
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
LIBLEPT_HEADERSDIR=/usr/local/include
./configure --with-extra-includes=/usr/local/include --with-extra-libraries=/usr/local/lib
make

I don't think this is related to Tesseract. The same sequence works for me on Debian Jessie and Stretch. Maybe a damaged local Leptonica build or installation?

 That cannot work of course.
You used LIBLEPT_HEADERSDIR. I did not find it in the code, so I think that is without any effect. The other options for configure, namely --with-extra-includes and --with-extra-libraries should not be needed. Setting PKG_CONFIG_PATH should be sufficient if you have installed code in /usr/local. I try to avoid such extra settings because they are confusing: which installation of Leptonica is used? Therefore I usually install in /usr (see my packages at https://digi.bib.uni-mannheim.de/tesseract/leptonica/).

I was following the instructions from the compiling page. I will try again just with PKG_CONFIG_PATH.

I have tried multiple times, different options, still getting same error.
Do not know whether it is because I am using the latest version of leptonica rather than 1.74.2.
@danbloomberg Has anything changed between 1.74.2 to latest code to cause this breakage?

I don't know what the breakage is.  In this thread I see the failure to load liblept in 3.05, which might have followed from the failure to find lept_free() for some reason.
@shreeshrii  you say it works for the master branch; for what version does it fail, and what is the failure?

Dan,
I had built leptonica with latest ocde from github (after even 1.74.4). I was able to build master branch of tesseract with it. However, when I tried building the 3.05 branch it failed with the error related to lept_free().
I now rebuilt leptonica 1.74.2 ( git reset --hard 395728da046016859864eeb99c7a48121acaeb74) and am able to build tesseract 3.05.
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -std=c++11   -o tesseract tesseract-tesseractmain.o libtesseract.la  -lrt -lpthread
libtool: link: g++ -g -std=c++11 -o tesseract tesseract-tesseractmain.o  ./.libs/libtesseract.a -L/usr/local/lib /usr/local/lib/liblept.so -lz -lpng12 -ljpeg -lgif /usr/lib/x86_64-linux-gnu/libtiff.so -lwebp -lopenjp2 -lrt -lpthread

make[2]: Leaving directory `/home/sanskrit/tesseract-3.05/api'
Making all in .
make[2]: Entering directory `/home/sanskrit/tesseract-3.05'
make[2]: Leaving directory `/home/sanskrit/tesseract-3.05'
Making all in tessdata

Another possibility is that the configure process is checking specifically for 1.74.2 ?

@Shreeshrii , please try these four commands:
pkg-config --cflags lept
pkg-config --libs lept
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pkg-config --cflags lept
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pkg-config --libs lept

Here is my output (only two commands, as I have no installation in /usr/local):
$ pkg-config --cflags lept
-I/usr/include/leptonica
$ pkg-config --libs lept
-llept

Your build output neither shows something like -I/usr/local/include/leptonica (for the C++ compiler) nor -L/usr/local/lib -llept, so I expect that you have a broken file lept.pc somewhere on your computer. The above test will check that.

$ pkg-config --cflags lept
-I/usr/local/include/leptonica
$ pkg-config --libs lept
-L/usr/local/lib -llept
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pkg-config --cflags lept
-I/usr/local/include/leptonica
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pkg-config --libs lept
-L/usr/local/lib -llept

lept.pc.in  is from Jan 2016, and lept.pc.cmake is from Apr 22, 2017.
Neither have changed since 1.74.2, which was May 19, 2017.
On Tue, Jun 20, 2017 at 11:30 AM, Shreeshrii<EMAIL_ADDRESS>wrote:

$ pkg-config --cflags lept
-I/usr/local/include/leptonica
$ pkg-config --libs lept
-L/usr/local/lib -llept
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pkg-config --cflags lept
-I/usr/local/include/leptonica
$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pkg-config --libs lept
-L/usr/local/lib -llept
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/1000#issuecomment-309847263,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP6mLPzjSF1578BlEDpRoP7yt0Bms9Ieks5sGA-vgaJpZM4N_xcO
.

Here is config.log from the build with 1.74.4 - which had the error.
config.log.txt

My 1.74.4 builds fine with autotools on linux.
I don't have a conftest.cpp file, and configure doesn't look for it.
Attaching my lept.pc (generated from lept.pc.in).
lept.pc.txt

@Shreeshrii, does the build work with my 3.05 (I just added a backported commit)?

Dan, Leptonica is building fine. The problem is in building tesseract 3.05 branch.
Stefan, I will try building with your 3.05 and report.
Thank you both for your prompt responses.

@stweil Tesseract part of the build happened correctly using your 3.05.
I am now getting similar error with make training.
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2 -std=c++11   -o text2image text2image.o libtesseract_training.la libtesseract_tessopt.la -licui18n -licuuc -licudata   -licuuc -licudata    ../api/libtesseract.la -licuuc -licudata   -lpango-1.0 -lpangocairo-1.0 -lgobject-2.0 -lglib-2.0 -lcairo -lpangoft2-1.0 -lfontconfig  -lpthread
libtool: link: g++ -g -O2 -std=c++11 -o .libs/text2image text2image.o  ./.libs/libtesseract_training.a ./.libs/libtesseract_tessopt.a -licui18n ../api/.libs/libtesseract.so -licuuc -licudata -lpango-1.0 -lpangocairo-1.0 -lgobject-2.0 -lglib-2.0 /usr/lib/x86_64-linux-gnu/libcairo.so -lpangoft2-1.0 -lfontconfig -lpthread
/usr/bin/ld: text2image.o: undefined reference to symbol 'pixGetHeight'
//usr/local/lib/liblept.so.5: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [text2image] Error 1
make[1]: Leaving directory `/home/sanskrit/tesseract-3.05/training'
make: *** [training] Error 2
make[1]: Entering directory `/home/sanskrit/tesseract-3.05/training'
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2 -std=c++11   -o text2image text2image.o libtesseract_training.la libtesseract_tessopt.la -licui18n -licuuc -licudata   -licuuc -licudata    ../api/libtesseract.la -licuuc -licudata   -lpango-1.0 -lpangocairo-1.0 -lgobject-2.0 -lglib-2.0 -lcairo -lpangoft2-1.0 -lfontconfig  -lpthread
libtool: link: g++ -g -O2 -std=c++11 -o .libs/text2image text2image.o  ./.libs/libtesseract_training.a ./.libs/libtesseract_tessopt.a -licui18n ../api/.libs/libtesseract.so -licuuc -licudata -lpango-1.0 -lpangocairo-1.0 -lgobject-2.0 -lglib-2.0 /usr/lib/x86_64-linux-gnu/libcairo.so -lpangoft2-1.0 -lfontconfig -lpthread
/usr/bin/ld: text2image.o: undefined reference to symbol 'pixGetHeight'
//usr/local/lib/liblept.so.5: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [text2image] Error 1
make[1]: Leaving directory `/home/sanskrit/tesseract-3.05/training'
make: *** [training-install] Error 2

Sorry, the fix for training/Makefile.am was missing. Updated now.

Thank you, @stweil. It was able to build 3.05 branch with the latest leptonica.
~$ tesseract -v
tesseract 3.05.01
 leptonica-1.74.2
  libgif 4.1.6(?) : libjpeg 8d (libjpeg-turbo 1.3.0) : libpng 1.2.50 : libtiff 4.0.3 : zlib 1.2.8 : libwebp 0.4.0 : libopenjp2 2.1.2

However, even though I have built the latest leptonica (1.74.4), version info is still reporting 1.74.2. So, you maybe right that there maybe some old  file somewhere ...
Is there a way to check the version info for leptonica, other than through tesseract -v?

Try this:
find /usr -name *lept*

You need to recompile Tesseract after you re-installed Leptonica

... or use ldd /usr/local/bin/tesseract to see which libraries are used. Maybe LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/tesseract -v gets the expected Leptonica version.

You need to recompile Tesseract after you re-installed Leptonica

In most cases this is not necessary, because typically Leptonica is not linked statically. That means Tesseract uses a shared library for Leptonica, and that library can be replaced by a newer one without harm (as long as there are no incompatible changes).

... (as long as there are no incompatible changes).

https://github.com/DanBloomberg/leptonica/issues/240

$ find /usr -name *lept*
/usr/share/doc/liblept4
/usr/share/doc/liblept5
/usr/lib/liblept.so.4.0.2
/usr/lib/liblept.so.4
/usr/lib/x86_64-linux-gnu/liblept.so.5.0.1
/usr/lib/x86_64-linux-gnu/liblept.so.5
/usr/local/include/leptonica
/usr/local/include/leptonica/leptwin.h
/usr/local/lib/liblept.so.5.0.1
/usr/local/lib/liblept.so.5
/usr/local/lib/liblept.la
/usr/local/lib/pkgconfig/lept.pc
/usr/local/lib/liblept.a
/usr/local/lib/liblept.so

$  ldd /usr/local/bin/tesseract
        linux-vdso.so.1 =>  (0x00007ffe1c126000)
        liblept.so.5 => /usr/local/lib/liblept.so.5 (0x00007f2d0d207000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2d0cfe9000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2d0cce4000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2d0c9de000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2d0c7c8000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2d0c3ff000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2d0c1e6000)
        libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f2d0bfc0000)
        libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f2d0bd6a000)
        libgif.so.4 => /usr/lib/x86_64-linux-gnu/libgif.so.4 (0x00007f2d0bb61000)
        libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f2d0b8ef000)
        libwebp.so.5 => /usr/lib/x86_64-linux-gnu/libwebp.so.5 (0x00007f2d0b696000)
        libopenjp2.so.7 => /usr/lib/x86_64-linux-gnu/libopenjp2.so.7 (0x00007f2d0b45b000)
        /lib64/ld-linux-x86-64.so.2 (0x000055d6a8161000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f2d0b238000)
        libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f2d0b02a000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2d0ae26000)

$ LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/tesseract -v
tesseract 4.00.00dev-549-g2b854e3
 leptonica-1.74.2
  libgif 4.1.6(?) : libjpeg 8d (libjpeg-turbo 1.3.0) : libpng 1.2.50 : libtiff 4.0.3 : zlib 1.2.8 : libwebp 0.4.0 : libopenjp2 2.1.2

So it's using the Leptonica from /usr/local which claims to be leptonica-1.74.2.

But I built using latest leptonica sources, config.log as well as allheaders.h have patch level as 4.
I will rebuild again and see.
Thanks!

I built leptonica again after doing make distclean , now I get the correct lib version. Thanks!
tesseract -v
tesseract 4.00.00dev-549-g2b854e3
 leptonica-1.74.4
  libgif 4.1.6(?) : libjpeg 8d (libjpeg-turbo 1.3.0) : libpng 1.2.50 : libtiff 4.0.3 : zlib 1.2.8 : libwebp 0.4.0 : libopenjp2 2.1.2

@shreeshrii  Is it worth doing a mini-postmortem on this?  Just a couple of paragraphs on what went wrong and what we might do to prevent this from happening again.

@DanBloomberg
I usually update leptonica by using the following script in the leptonica directory.
#!/bin/bash
git pull origin
./autobuild
./configure --disable-dependency-tracking
#./configure
make
sudo make install
sudo ldconfig

I follow a similar build script for tesseract master branch for building 4.0.
Based on an error report in the forum, I tried building 3.05.01 and received the error related to lept_free (reported above).
As part of the experimentation, I did a hard reset in leptonica directory at one time to get to the commit for 1.74.2 and built leptonica and tesseract 3.05.01 after that without any problem.
@stweil then backported a couple of commits to tesseract 3.05.01 branch to fix the problem regarding lept_free() and I rebuilt leptonica with the latest source and then built tesseract 3.05.01. It built without the lept_free error.
It is possible that I did not build leptonica using the script which does autobuild and configure, but rather just did
git pull origin
make
sudo make install

I also built tesseract 4.0 from master branch after that. I noticed in both of these, that leptonica version was reported as 1.74.2.
After following Amit and Stefan's suggestions to see whether I had multiple versions of files causing the older version to display, I decided to rebuild leptonica again. This time I did make distclean in leptonica directory before running the build script.
Then without needing to rebuild tesseract, tesseract -v showed that it was using leptonica 1.74.4.
@amitdo So, as mentioned by Stefan, it is not necessary to rebuild tesseract to use the newer leptonica library.

@shreeshrii
Thanks!  Very nice summary.
We have several ways to build tesseract and the associated libraries.
An embarrassment of riches, as the French would say (except
in their own way).
It seems to me that the current suggestions in the leptonica
README.html for building with autotools are still valid.
It also references the documentation in
https://github.com/tesseract-ocr/tesseract/wiki/Compiling
and I see that you updated that page 2 weeks ago.  Would
you make any further changes there?
On Wed, Jun 21, 2017 at 12:08 PM, Shreeshrii<EMAIL_ADDRESS>wrote:

@DanBloomberg https://github.com/danbloomberg
I usually update leptonica by using the following script in the leptonica
directory.
#!/bin/bash
git pull origin
./autobuild
./configure --disable-dependency-tracking
#./configure
make
sudo make install
sudo ldconfig
I follow a similar build script for tesseract master branch for building
4.0.
Based on an error report in the forum, I tried building 3.05.01 and
received the error related to lept_free (reported above).
As part of the experimentation, I did a hard reset in leptonica directory
at one time to get to the commit for 1.74.2 and built leptonica and
tesseract 3.05.01 after that without any problem.
@stweil https://github.com/stweil then backported a couple of commits
to tesseract 3.05.01 branch to fix the problem regarding lept_free() and I
rebuilt leptonica with the latest source and then built tesseract 3.05.01.
It built without the lept_free error.
It is possible that I did not build leptonica using the script which does
autobuild and configure, but rather just did
git pull origin
make
sudo make install
I also built tesseract 4.0 from master branch after that. I noticed in
both of these, that leptonica version was reported as 1.74.2.
After following Amit and Stefan's suggestions to see whether I had
multiple versions of files causing the older version to display, I decided
to rebuild leptonica again. This time I did make distclean in leptonica
directory before running the build script.
Then without needing to rebuild tesseract, tesseract -v showed that it
was using leptonica 1.74.4.
@amitdo https://github.com/amitdo So, as mentioned by Stefan, it is not
necessary to rebuild tesseract to use the newer leptonica library.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/1000#issuecomment-310176024,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP6mLD_tbdPGJyKbdREo7_bUNgEsi53Sks5sGWozgaJpZM4N_xcO
.

he wiki pages are public and have been updated by different people at different times.  I did update the Compiling wiki page with the needed leptonica version recently.
There are multiple Install/Compiling instructions.
I think that https://github.com/tesseract-ocr/tesseract/blob/master/INSTALL.GIT.md is better.
I will try to streamline the compiling page - maybe delete duplicated info and refer to the installation page instead. However, it will require review by @stweil @egorpugin and @zdenop .

Actually, the page I referred to looked pretty good, and I was just
referring to the small section on constraints for compiling leptonica.
On Fri, Jun 23, 2017 at 3:39 AM, Shreeshrii<EMAIL_ADDRESS>wrote:

he wiki pages are public and have been updated by different people at
different times. I did update the Compiling wiki page with the needed
leptonica version recently.
There are multiple Install/Compiling instructions.
I think that https://github.com/tesseract-ocr/tesseract/blob/master/
INSTALL.GIT.md is better.
I will try to streamline the compiling page - maybe delete duplicated info
and refer to the installation page instead. However, it will require review
by @stweil https://github.com/stweil @egorpugin
https://github.com/egorpugin and @zdenop https://github.com/zdenop .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/1000#issuecomment-310633374,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP6mLIm9-ICPyTO9xz1Q9Iywm5uT3W06ks5sG5XfgaJpZM4N_xcO
.

I seem to be hitting the problem too - trying to add in a cross compile for powerpc into our travis build setup, building leptonica first and then tesseract and I seem to be hitting this same issue no matter what I try, if I use <IP_ADDRESS> download. If I pull latest straight from git, it compiles fine. Also builds fine for android, mac, windows, probably even native ppc.
@Shreeshrii you mention some backports to <IP_ADDRESS>, could you help with any details on those? I looked through the commit log but from a quick look from around Jun I didn't spot anything immediately obvious. I'm wondering if its something I can apply as a patch, just that pulling a non-release version usually ends up with headaches down the road

git clone -q --branch=3.05 https://github.com/tesseract-ocr/tesseract.git tesseract-3.05

This will fetch https://github.com/tesseract-ocr/tesseract/tree/3.05, but the latest 3.05.x is 3.05.1: https://github.com/tesseract-ocr/tesseract/tree/3.05.01
Currently, version 4.0 still has the legacy engine from 3.0x (plus the new LSTM engine).
It can be used with the 3.05/3.04 traineddata files (--oem 0 or --oem 3).


str2pcap should just parse debug logs

Rather than copy & pasting lines, just parse them.

Can you add more information on this? I see contents being copied from inputfile to output file in str2pcap/main.go. What do you mean by "JUST" parsing? Do you want to remove the code that writes to outfile?

right now the --debug flag spits out the hex values of the packet.  Currently to use str2pcap someone has to copy and paste those values from the output and into a file which obviously sucks.  I believe the logger can output JSON which would make parsing the logs automatically pretty simple for str2pcap to do without requiring the user to copy  & paste.

no need now that we create pcap files


add examples 10 & 11 to index

Add examples 10 and 11 to the demo index

Well, well.. that drive-by-changes aren't good.
The actual change (missing chapters) is landed as 36402de right now, but the other stuff not.
Your 36df7e4 (and fixup 46f9228 and c367717)contains a new feature I'm not sure why we should need this and another (interesting) demo for firebase.
And 12f0a06? Heh? What's that? :)

oh, there have been more files added ... yap, I agree with @knalli . Also, the commit comment format should be conforming to standards... also these special comments are really hard to interprete as an "external" for this PR...

Closing due inactivity.


Netlify Adapter: Custom redirects not working

Describe the bug
Custom Netlify redirects don't work. Neither using netlify.toml nor static/_redirects.
To Reproduce

Setup a minimal SvelteKit app using the Netlify adapter
Specify a redirection rule, e.g. put /old /new in static/_redirects
Deploy the build output on Netlify (or test it locally using Netlify Dev)
Navigate to https://YOUR-TEST-PAGE.example/old
Now you'll get a Svelte 404 error

Expected behavior

Instead of the 404, my website should actually redirect to https://YOUR-TEST-PAGE.example/new
Document that redirections in netlify.toml aren't supported (for now?) because the adapter itself defines a "catch all" in _redirects making all in netlify.toml unusable

Information about your SvelteKit Installation:

@sveltejs/kit: next => 1.0.0-next.64
@sveltejs/adapter-netlify => 1.0.0-next.4
svelte: ^3.29.0 => 3.36.0
vite: ^2.1.0 => 2.1.4

Severity
For me personally? High, as I'm planning to replace an old website with SvelteKit and have to gracefully deal with old URLs. At least the _redirects definitions should work.
Additional context

I already noticed that the adapter is simply overwriting my own _redirects file with its own content because writeFileSync() is used.
When investigating the issue locally I tried to change it to appendFileSync() (which makes more sense in my opinion).
Then however I'm getting the following:Must use import to load ES Module: MY-PROJECT/functions/render/index.js
require() of ES modules is not supported.
require() of MY-PROJECT/functions/render/index.js from /npm/node_modules/netlify-cli/node_modules/lambda-local/build/lambdalocal.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from MY-PROJECT/package.json.

Renaming index.js to index.cjs as suggested leads to another error.

Here I had to stop because I don't know what that error above exactly means. I hope my report is helping though. Nice project btw!

Thinking about it a little more, doing the following seems to be best:

Append the required line to an existing _redirects rather than overwriting it.
Add a flag to disable adding this redirection altogether so we can use the more powerful netlify.toml instead (and have to add it there ourselves).
Document this special treatment in the readme.

What do you think?

I had a good night's sleep as well and a clearer understanding of the situation after reading more into the Netlify docs. I agree with your latest comment, using netlify.toml should be the preferred way to go.
One addition I suggest to requirements:

If there is no existing _redirects file and the netlify.toml does not have the required redirect, we keep the current behaviour (i.e. we create the _redirects file with /* /.netlify/functions/render 200

Thought about it even more. ;) We should try to make it even smarter:

Check if the adapter-specific redirection is in the TOML:

Yes: nothing to do here anymore, proceed normally.
No: continue.

Check if there are any other [[redirects]] in the TOML:

Yes: fail with an error, e.g. "Netlify adapter's redirection rule is missing. Please add [...] at the end of your netlify.toml"
No: continue.

Do the appendFileSync(publish + '/_redirects', '\n\n/* /.netlify/functions/render 200') thing. It should automatically create the file if it doesn't already exist. Else, \n ensures that it's added in a proper new line.

Additionally we should point out in the readme that no (other) custom "catch all" redirects are allowed, neither in netlify.toml nor in _redirects.

These steps sound reasonable to me. I can adjust the open PR #932 accordingly, but wonder if @benmccann had any comments before I start to work on the changes.
For anyone else ending up here, I needed a workaround to support existing redirects (commit). Here's what's needed:

In netlify.toml, add the following redirect rule at the end of the file:[[redirects]]
  from = "/*"
  to = "/.netlify/functions/render"
  status = 200

Delete the ./build/_redirects file once svelte-kit build completes

I'm not that familiar with netlify.toml. I'd say if you both agree on what to change then go ahead and send that PR

I just noticed that @Rich-Harris made some changes. So now a custom _redirects isn't even considered anymore (just the internal file is used). So how should we proceed with this?
I still like the approach described above https://github.com/sveltejs/kit/issues/930#issuecomment-817077432 as I have a rather long _redirects that would be less maintainable using the netlify.toml format. However step 3 would have to be adjusted somehow... Any suggestions?

I think we should respect the existing config. See https://github.com/sveltejs/kit/issues/60 which is an open issue for that

We can close the PR as it already diverged too much (and the changes wasn't that big anyway).
To be honest I'm unsure how we should proceed here. I hoped @Rich-Harris would chime in and discuss with us how it's supposed to be done now because with the recent changes the possibility to easily fix it got worse: now _redirects is simply overridden not taking an existing _redirects into account at all.

Is there a reason you even need to define custom redirects in netlify.toml? You can do it in SvelteKit itself either in load or handle. Doing it in SvelteKit would make it much easier to migrate to another host should you ever have a desire to do that

@benmccann but that wouldn't work for a static site – unless I've misunderstood what you're suggesting?
Surely the simple solution is for SvelteKit to append (or prepend) to _redirects rather than overwriting it. Would that be difficult?

Oh, I may have misunderstood – do the hooks you're suggesting using run onload, in the browser?
But in that case you would have the overhead of the client loading the page before the redirect, which is far from ideal, plus you can't send eg a 301 redirect which is useful for search engines.

Good point. I think that it could do a client-side redirect, but that's probably different than the server-side redirect that I imagine that Netlify would do
It seems very reasonable update the file rather than overwriting it. It's probably not something the maintainers would get to, but I'd be open to a PR from the community for it

I will need this when I use SvelteKit in production so I'd be happy to take a look – but I probably won't have time in the near future, so if anyone else wants to get to it before me, go ahead! 😄
Regarding doing it in SvelteKit so that migrating to another host in future is easier – I wonder if a good approach would be to add the redirects to SvelteKit's config, and then the individual Adapters (Netlify, Vercel, etc) could output them in the appropriate format for each host? Of course that's more work to implement, but it would give greater portability.

I need it too so I started a quick PR to get some progress on this: https://github.com/sveltejs/kit/pull/1484
btw, I decided against the more robust implementation described in https://github.com/sveltejs/kit/issues/930#issuecomment-817077432 as it's overengineering and the "quick fix" -- just appending the specific rule -- should do it too.

Thanks @kvn-shn, that PR looks reasonable – though I don't know if the fatal error is really necessary if a catch-all is found in netlify.toml? Maybe just a warning that it won't do anything?
Thinking more about what @benmccann said – and looking further into how the Netlify Adapter works – I realise that a site published with the Netlify Adapter is not static / pre-rendered after all, but server-rendered using Netlify Functions.
That being the case, perhaps the handle hook would work for this after all.
That said, I still think the adapter shouldn't break _redirects unnecessarily, so I'm +1 on the solution in #1484.

Thanks for your feedback. In my opinion it's reasonable to throw an error because for the Netlify Adapter it's simply invalid to have any other catch-all redirect besides the generated one.
One benefit of using Netlify's redirects instead of something SvelteKit provides is that they are way more powerful, e.g. redirects by country or language or role, setting headers etc.

Both good points, and bearing that in mind the PR looks good to me. 👍

One more thought: bearing in mind that netlify.toml supports more advanced redirect support (including all the syntax described in the page you linked), and bearing in mind that a catch-all in _redirects prevents any redirects in netlify.toml from working, is there a reason Netlify Adapter's catch-all can't go in netlify.toml instead of in _redirects?

Netlify's redirects aren't and can't be more powerful than SvelteKit's. SvelteKit has access to all the same info as Netlify and potentially more and allows you to write custom code. If you write something in handle or load it will be run as a Netlify function and you can still prerender pages if you'd like

@benmccann true, but it will require several lines of code to do what you could do with one line in _redirects or with a simple netlify.toml config. It's probably faster too. For the same reasons, on my own server I do redirects in nginx config instead of in application code.
I guess it comes down to personal preference, but either way, I think in principle the adapter shouldn't break Netlify's built-in functionality unless it really has to.


Parser error with mixed type and value imports

Since some time kit generates a tsconfig.json with some defaults and forces us to extend from it (https://github.com/sveltejs/kit/blob/60b3e9e9f489befe64d95afc812b1fcb921563e2/packages/kit/src/core/sync/write_tsconfig.js#L82). This file mainly sets
		"importsNotUsedAsValues": "error",
		"isolatedModules": true,
		"preserveValueImports": true

which means types must be imported with import type. The example seems outdated in this regard and just uses a normal value import.
While writing queries/mutations/fragments vscode helpfully suggests the correct imports and I end up with something like
   import { fragment, graphql } from '$houdini';

After writing the gql string I generate and try to add the type and vscode insists on generating
  import { fragment, graphql, type GetMyCartItems } from '$houdini';

ESLint is happy but now preprocessing fails with (originating from the preprocess-esm/index.js file)

Error while preprocessing /.../src/routes/warenkorb.svelte - Unexpected token, expected "," (3:26)

The workaround is simple:
  import type { GetMyCartItems } from '$houdini';
  import { fragment, graphql } from '$houdini';

but I did this so often I can no longer be bothered with this...
Could this be related to the second parsing phase with babel?

Hm, interesting. I'll try to find some time to dig into this to see what could be causing it. Babel is configured to parse the file with the typescript parser so I would be surprised if it's causing the issue 🤔

Just bumped into this as well running npx houdini init. Error was:
Error: Encountered error parsing src/lib/shared/components/calendar/Calendar.svelte: Unexpected token, expected "," (2:22)

The line of code indicated by the error:
  import dayjs, { type Dayjs } from 'dayjs';

Column 22 is the end of the type keyword so I refactored it to:
import dayjs from 'dayjs';
import type { Dayjs } from 'dayjs';

And it resolved the error.


reason task only shows used classes if there are no exposed classes

Plugin version
1.32.0
Gradle version
8.6
JDK version
17
(Optional) Kotlin and Kotlin Gradle Plugin (KGP) version
1.9.23
(Optional) Android Gradle Plugin (AGP) version
8.3.2
Describe the bug
I'm running the reason task to understand why modules depend on our hairball module. It will only output used classes if there are no exposed classes. Is this intended?
To Reproduce
(unfortunately I have to redact specific class/module information)
With an exposed class:
$ ./gradlew :feature-library:xxx:reason --id :hairball

> Task :feature-library:xxx:reason

----------------------------------------
You asked about the dependency ':hairball'.
You have been advised to change this dependency to 'api' from 'implementation'.
----------------------------------------

<clip>

Source: debug, main
-------------------
* Exposes 1 class: com.pinterest.hairball.xyz (implies api).

When I comment out the usage of the exposed class:
$ ./gradlew :feature-library:xxx:reason --id :hairball

> Task :feature-library:xxx:reason

----------------------------------------
You asked about the dependency ':hairball'.
There is no advice regarding this dependency.
----------------------------------------

<clip>

Source: debug, main
-------------------
* Uses 11 classes, 5 of which are shown: com.pinterest.xxx ... <clipped>

Expected behavior
I'd like to use this task to prioritize classes to pull out of our hairball module, but without complete information it's difficult. Is there a way to make the reason task print out everything?

Thanks for the issue! I think that was an intentional decision, but I'm not sure it's well-justified.

Yeah it seemed intentional when I looked at the code, and certainly my use case may be slightly different than what the plugin was designed to do. Perhaps if I get some time I'll see if it's possible to put this behavior behind a flag and put up a PR.

I don't know that it needs a flag. If you wanted to contribute, I think just showing both exposed and used together would be ok. Note that the list can get quite long, so I think there's already code that limits the number of classes displayed. If you wanted to see the full list, that might be something where we'd want a flag.


[Realm Browser Plugin] Add Sanity check to alert user when more than file is being opened

This PR is a sequel from #2171. As suggested that, plugin should at least warn user in the case where more than 1 file is detected and about to open. This PR aims to fix this.
@TimOliver, I were trying several things, but I got no luck. :crying_cat_face:
Using [[NSWorkspace sharedWorkspace] launchApplicationAtURL:self.browserUrl options:0 configuration:configuration error:&error]. or replacing with options:NSWorkspaceLaunchDefault seems to work, but it fails to open Realm File on the subsequent runs (meaning that closing the Realm file window without quitting the Realm Browser and go to File > Open Realm ... in order to open Browser with Realm file again)
I am not familiar enough with Apple Script so unfortunately I couldn't make the [NSWorkspace openURLs:withAppBundleIdentifier:options:additionalEventParamDescriptor:launchIdentifiers:] to work.
:(
So, I decided that to the least having a check from plugin's standpoint is doable from my side. I went ahead and implemented those quickly in this PR. If you have a smarter way for this, I am all ears.

Hi @kittinunf! Thanks a lot for your follow-up pull request!
Ahh, okay then. It looks like we've ironically got just the wrong set of API combinations to make that possible. That's a bit of a shame. Sorry to have made you go away and research that! I might keep it in mind and see if there's a better way to do it (Even if it means some kind of direct communication between the plugin and the Browser).
I reviewed your code, and that looks like the best solution at present to me. That'll cover the portions of the multiple file handling operations that weren't getting covered on Realm Browser's side.
@jpsim I'm happy with this pull request. Shall we merge it in?


Created output_html.py. Added output html option to main.py.

Created pyflowchart/output_html.py to store three variables used writing output html file.
Modified pyflowchart/main.py to import these three variable and import os.
import os is needed to convert a path to the output file into a output filename used as the header in the output html file.
This is a very basic output entirely based on example.html from github flowchart.js example and flowchart.js.org . I added a helper button with hidden default text based on helper text from flowchart.js.org for tweaking the output html file for live and or saved changes.

Hi so,
I've revised variable names and figured out I can use """ to block quote raw html. So replaced string I had read in with python to get it formatted safe to this version. I added more comments on what I'm trying to do in both main and in this output_html.py python file. Does that resolved your concerns?

Thanks for the example. I'm fine moving it all into separate python file. reason i get variables in separate file is I didn't want to add a lot to main.py. function probably like
output_html(file_name: str, html_title: str, flowchart: str)
In main how do you feel about
if bool(output):
if bool(field):
f.write(field)
output_html(file_name=output, html_title=field, flowchart=flowchart.flowchart())
else:
f.write()
output_html(file_name=output, html_title=os.path.basename(f.name), flowchart=flowchart.flowchart())

Tried to amend a typo in commit comment and ended up where my branch is now. But otherwise, I've implemented as we discussed last night.

A little 🐛.

I have resolved it.

Should we have
with open(os.path.expanduser(output_name), 'w') as f:
instead of
with open(output_name, 'w') as f:
?

Should we have
with open(os.path.expanduser(output_name), 'w') as f:
instead of
with open(output_name, 'w') as f:
or even
with open(os.path.expandvars(os.path.expanduser(output_name)), 'w') as f:
?

Yes. While I am not used to write this, it's not hurt anyway. (Btw is this a common pattern? I didn't see this often.)

What I am concerned now is that we'd better check the suffix of file_name first. In case that users entering commands with -o notsupported.png, which may cause misunderstanding. We should only accept the .html suffix. How do you like this?

It broken into separate sections. As I've written it, if user does something convoluted it'll render a bad path.
i.e. ~%username%\ will give the same bad result with
os.path.expandvars(os.path.expanduser('~%USERNAME%')) or
os.path.expanduser(os.path.expandvars('~%USERNAME%'))
as the expansions are mutually exclusive. but combining will give you correct results for ~\file.bob ~/file.bob %temp%\file.bob %temp%/file.bob
pallet's click handles it this way:
https://github.com/pallets/click/blob/388b0e14093355e30b17ffaff0c7588533d9dbc8/src/click/utils.py#L575-L624
for arg in args:
      if user:
          arg = os.path.expanduser(arg)

      if env:
          arg = os.path.expandvars(arg)

html or htm sounds fine to me we can append if they don't include that suffix. We can output a warning if they didn't provide letting them know we changed suffix. Would you like me to implement?

I have written lines of code for this (but not completed). Would you mind it if I implement this feature and push it to your branch?

Looking at click the handling of args is broken into separate sections. As I've written it, if user does something broken it'll render a bad path. i.e. ~\%username%\ will give the same bad result with os.path.expandvars(os.path.expanduser('~\%USERNAME%')) or os.path.expanduser(os.path.expandvars('~\%USERNAME%'))
The two expansions are mutually exclusive. Combining will give you correct results for ~\file.bob ~/file.bob %temp%\file.bob %temp%/file.bob or %userprofile%/file.bob as some examples. pallet's click handles it this way: https://github.com/pallets/click/blob/388b0e14093355e30b17ffaff0c7588533d9dbc8/src/click/utils.py#L575-L624
for arg in args:
      if user:
          arg = os.path.expanduser(arg)

      if env:
          arg = os.path.expandvars(arg)

Understood. Thanks for the explanation.

html or htm sounds fine to me we can append if they don't include that suffix. We can output a warning if they didn't provide letting them know we changed suffix. Would you like me to implement?

I implemented it in https://github.com/ddfault/pyflowchart/commit/56bf90bb4bdb99ea40cdb5346dd1b95fbfb7baa1. Err, I am sleepy and not sure if I made it correctly. Please review it.

My gripes are we don't append the extension we need, don't tell them it needs to end in a specific extension, and don't output to screen given a bad -o string
py -m pyflowchart %path%\code.py -f bob -o code.bob only gets me
Error: not supported output file format:', code.bob
I'm not telling them in the helper text
parser.add_argument('-o', '--output', default="", type=str, help="Output HTML file with flowchart")
that they need to specifcy .html or .htm
I would rather just append the .htm or html for them in this case. We could check for known image types and then error out.
or
print(f'\n*** Error: {ext} not supported output file format:', file_name)  # TODO: stderr
print(f'\n*** Error: {ext} not supported output file format. Currently only .htm or .html supported:', file_name)  # TODO: stderr
something along those lines is what I can think of atm.

My gripes are we don't append the extension we need, don't tell them it needs to end in a specific extension, and don't output to screen given a bad -o string
py -m pyflowchart %path%\code.py -f bob -o code.bob only gets me
Error: not supported output file format:', code.bob
I'm not telling them in the helper text
parser.add_argument('-o', '--output', default="", type=str, help="Output HTML file with flowchart")
that they need to specifcy .html or .htm
I would rather just append the .htm or html for them in this case. We could check for known image types and then error out.
or
print(f'\n*** Error: {ext} not supported output file format:', file_name) # TODO: stderr
print(f'\n*** Error: {ext} not supported output file format. Currently only .htm or .html supported:', file_name) # TODO: stderr
something along those lines is what I can think of atm.

I see. However I’m a bit disagree with your idea of appending an extension to the file name automatically.
As an UNIX user, I would not be happy to see a program writing to a file whose name is not explicitly mentioned by me.. I don’t know how to explain this. I just feel this behavior is not in common. “I commanded you to write a file called foo, but you wrote to the foo.html” this is a bit strange. It makes me feel like I’m losing my control of the computer.
My point is that I implemented the output function in a way that supports any format that might be implemented in the future.
In this case, the original print(flowchart) and our new output_html() are becoming two of the possible formats that we support, though they are the only two formats that are available now.
So maybe an amendment to the help str of -o could explain my design clearer: “Output the flowchart to specific file with a format indicating by the extension name.”
How do you think?

I edited again after what your comment shows as your email. Sorry I keep editing my comments :/
Your approach is probably the simplest and gets the point across (versus appending and checking for known image extensions) not outputting to console on error just requires them tot run again which isn't a big deal so probably right move.
I'd still would like the error to say something along the lines of:
print(f'\n*** Error: {ext} not supported output file format. Currently only .htm or .html supported:', file_name)  # TODO: stderr
Also by the way... I just pushed HTML changes. You can now open the html and download the rendered flowchart as png or svg. My flowcharts always need a bit of tweaking no,left yes,right etc before they look clean. The download links update when you click run and generate a new canvas.

Downloading the image is super useful! (Btw I know a package that may make this easier: https://www.npmjs.com/package/html-to-image)
And I amended the help str of -o as well as the error message as discussed above.
I feel it's time to merge this PR. Thanks for all of your contribution.
Would you find anything to be done before we merge it?

Sure. I'll take a look at revising the readme tomorrow and then you can revise from there.

I've pushed README.md changes. Please review and revise as you see fit.

I've pushed README.md changes. Please review and revise as you see fit. I haven't updated the TODO section or the non-english README.md

okk, thanks

I made some change on both code & docs:

export the output_html function and move your file doc of output_html.py as output_html()'s function doc to support:

>>> from pyflowchart import output_html
>>> help(output_html)

refactor the document about output html.

Any comments or suggestions?

Thanks for your contributions, @ddfault! I'm merging this now. Please feel free to continue improving it in the future.


IGUANA sends repeated request with chunked encoding

Happens when queries are sent via post request and are cached. Expected behavior should be, that repeated request stay non-chunked.

Workaround is to change the HTTP protocol version to 1.0.


How to handle disabled application in yafti, but expected by bluefin

It could happen for example that a user disables Blackbox terminal on first run, because why do I need another terminal? Which is totally understandable. But can lead to unexptected behavior.
The problem may occur that we inside bluefin point shortcuts to blackbox because we expect it to be installed, which then would not work anymore. Is this something we need to fix? And if so, what would be the best way?

Maybe in the description we could mention that it's necessary for development or something? We could also just switch to the RPM?

Oh that would be an easy solution indeed. Just add (Expected) or something after the package name. And say in the description. That expected package by bluefin are marked with (Expected) in the description.

How about moving the core applications to a seperate screen like this:

And after this configure the other applications:

Yeah sounds good to me!


Field String Regex Validation

I think regex validation would be a very nice feature for this action. Not sure how easy it is to implement this.
For example, the following schema file
#schema.yaml
dev:
  - type: string
  - regex: arn:aws:s3:::[a-zA-Z0-9_\-]+\/\*

Should throw an error when validating this file
#file.yaml
dev:
  - something-here # This would fail

It might be rather difficult, as the changes would have to be made upstream to the dependency that this Action uses for YAML validation -> https://github.com/ketanSaxena/schema-validator

Yes I also think so. I would still like to try and come up with a proposal implementation for this, since I think it would make the tool very powerful.

I absolutely agree! Feel free to link back to this issue if you happen to work on this upstream and if anything lands please let me know as well since that would be awesome.
I'll close this out for now though as there isn't anything for us to do in this project


Create new BB: Subscription Form

This is a Building Block for our upcoming News/Magazine Kit. For anyone who wants to contribute some building blocks to the site, feel free to comment below to let us know you're taking this one on!
Description/Interaction: Just a simple subscription form. Should include a header, a subheader, and an email field.
Example:

Hi,
I would like to take a look at this one. Let me know if that's cool.

Go for it @AllyReid

added PR https://github.com/zurb/building-blocks/pull/91

@AllyReid Merged! Thanks!


Test enhancement for assertion call and usages

Changed log

Using the * to let all branch name can be triggered on GitHub Actions.
Remove $settings variable because it's assigned, but not used.
Using the assertNull to assert expected is null.
Using the assertFalse to assert expected is false.
Using the assertTrue to assert expected is true.
According to the PHPUnit fixtures reference, the setUp method is protected function setUp(): void and the tearDown method is protected function teaDown(): void.

thanks!


avoid crash when no bio is available

Right now, if a profile has no "bio" available, Pnytter crashes. This fixes it.

Nice catch! Could you add a test for this case? If not I will

I don't know why the test workflow is not running on this PR, it doesn't even let me give permission for its execution. I've opened a question here about it.

I merged it although some tests were failing, will fix them later. Regards and thanks!


feat: add decorator to decrypt sessions outside requests

Fixes #52
Hi, I was trying to figure out an easier way to get access to the session inside of fastify-websocket handlers.
Since wsHandlers don't have access to the complete FastifyRequest object, you don't have direct access to the session and the only way to do this is to parse the cookie from the headers and lookup the session manually.
I'd love to hear some feedback on this approach.

Coverage decreased (-100.0%) to 0.0% when pulling 1e153b0ad0e758b9557a486e6738af853a134899 on DRoet:feat/decrypt-session into 16cae77e97508830db45dc9152606c5f31e68f80 on SerayaEryn:master.

Hmm I'm not sure why the coverage is failing, I still get 100% locally with the added tests

Awesome! Any plans for a 5.x.x release soon?

@DRoet Nice job! I didn't have the time yet to try this approach, but this makes it a lot easier!

@SerayaEryn My hands are itching to try this now, so I'd like to second @DRoet his question for a 5.x.x release 👍

v5.2.0 has been released.


Update documentation for Azure Stack Hub sample

Based on the feedback, this PR is to update documentation for Event Processor sample to use an alternative storage version as checkpoint store applicable to Azure Storage Hub stack.
cc @jfggdl

@jfggdl - we do have this sample listed in our readme - https://github.com/Azure/azure-sdk-for-java/pull/9170/files#diff-475ddf75cc250913b49bea28e7dcaacaR66


Cannot create or run existing flutter project

Unabl to run existing flutter projects and can't create a new one
This is what I get when running flutter doctor
`[☠] Flutter (the doctor check crashed)
    ✗ Due to an error, the doctor check did not complete. If the error message
      below is not helpful, please let us know about this issue at
      https://github.com/flutter/flutter/issues.
    ✗ Exception: Could not determine engine revision.
    • #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
      #1      Cache.engineRevision (package:flutter_tools/src/cache.dart:447:7)
      #2      FlutterVersion.engineRevision
      (package:flutter_tools/src/version.dart:152:46)
      #3      FlutterVersion.engineRevisionShort
      (package:flutter_tools/src/version.dart:153:55)
      #4      FlutterValidator.validate
      (package:flutter_tools/src/doctor.dart:458:75)
      #5      asyncGuard.<anonymous closure>
      (package:flutter_tools/src/base/async_guard.dart:111:32)
      #6      asyncGuard.<anonymous closure>
      (package:flutter_tools/src/base/async_guard.dart:109:18)
      #7      _rootRun (dart:async/zone.dart:1426:13)
      #8      _CustomZone.run (dart:async/zone.dart:1328:19)
      #9      _runZoned (dart:async/zone.dart:1861:10)
      #10     runZonedGuarded (dart:async/zone.dart:1849:12)
      #11     runZoned (dart:async/zone.dart:1780:12)
      #12     asyncGuard (package:flutter_tools/src/base/async_guard.dart:109:3)
      #13     Doctor.startValidatorTasks
      (package:flutter_tools/src/doctor.dart:205:9)
      #14     Doctor.diagnose (package:flutter_tools/src/doctor.dart:309:47)
      #15     DoctorCommand.runCommand
      (package:flutter_tools/src/commands/doctor.dart:50:48)
      #16     FlutterCommand.verifyThenRunCommand
      (package:flutter_tools/src/runner/flutter_command.dart:1320:12)
      <asynchronous suspension>
      #17     FlutterCommand.run.<anonymous closure>
      (package:flutter_tools/src/runner/flutter_command.dart:1161:27)
      <asynchronous suspension>
      #18     AppContext.run.<anonymous closure>
      (package:flutter_tools/src/base/context.dart:150:19)
      <asynchronous suspension>
      #19     CommandRunner.runCommand (package:args/command_runner.dart:209:13)
      <asynchronous suspension>
      #20     FlutterCommandRunner.runCommand.<anonymous closure>
      (package:flutter_tools/src/runner/flutter_command_runner.dart:281:9)
      <asynchronous suspension>
      #21     AppContext.run.<anonymous closure>
      (package:flutter_tools/src/base/context.dart:150:19)
      <asynchronous suspension>
      #22     FlutterCommandRunner.runCommand
      (package:flutter_tools/src/runner/flutter_command_runner.dart:229:5)
      <asynchronous suspension>
      #23     run.<anonymous closure>.<anonymous closure>
      (package:flutter_tools/runner.dart:62:9)
      <asynchronous suspension>
      #24     AppContext.run.<anonymous closure>
      (package:flutter_tools/src/base/context.dart:150:19)
      <asynchronous suspension>
      #25     main (package:flutter_tools/executable.dart:94:3)
      <asynchronous suspension>

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/sheheryartariq/Library/Android/sdk
    • Platform android-32, build-tools 32.0.0
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7772763)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.4)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.10.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7772763)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.20.0

[✓] Connected device (1 available)
    • Chrome (web) • chrome • web-javascript • Google Chrome 102.0.5005.61

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.`

https://github.com/flutter/flutter/blob/1a7a766de5937cd66a8f32834d5879cdcf95c9c3/packages/flutter_tools/lib/src/flutter_cache.dart#L447
https://github.com/flutter/flutter/blob/1a7a766de5937cd66a8f32834d5879cdcf95c9c3/packages/flutter_tools/lib/src/version.dart#L152
Looks like user does not have engine revision and seems to be on Mac.
Could it be similar to https://github.com/flutter/flutter/issues/90092?
Labeling it for further insights from the team.
/cc @christopherfujino

@Shery573 so your cache is in a bad state. I'm wondering, how did you get the Flutter SDK? (Did you download the prepackaged zip from the website? git clone from github? flutter upgrade from a previous version?)
In order to get back into a good state, you can delete your entire Flutter cache, which should be a command like (substitute your actual file path):
rm -rf $FLUTTER_SDK/bin/cache


Upgrade Datasette Docker to Python 3.11

Related:

#1768

I think this base image looks right: 3.11.0-slim-bullseye

I ran a very crude benchmark on my laptop using Locust (against the official macOS packages from www.python.org for Python 3.10 and Python 3.11) and saw a substantial speed increase:
533.89 requests/second on 3.11
413.56 requests/second on 3.10
That was from running Locust against this locustfile.py:
from locust import HttpUser, task

class CounterOne(HttpUser):
    @task
    def hello(self):
        self.client.get("/-/static/app.css")

Using:
locust --headless --users 4 --spawn-rate 4 -H http://<IP_ADDRESS>:8001

Ran the upgrade on the Datasette Cloud image first, works fine there.
https://simon.datasette.cloud/-/versions shows me:
{
    "python": {
        "version": "3.11.0",
        "full": "3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"
    },

Also tested by running this locally:
datasette publish cloudrun fixtures.db --service issue-1853

https://issue-1853-j7hipcg4aq-uc.a.run.app/-/versions now shows Python 3.11.0.

And tested datasette package like this:
datasette package fixtures.db -t datasette-package-python-upgrade-3-11

Then:
docker run -p 8081:8001 datasette-package-python-upgrade-3-11

And tested it like this:
curl http://localhost:8081/-/versions.json | jq

Output:
{
  "python": {
    "version": "3.11.0",
    "full": "3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"
  },

This broke the deploy of https://latest.datasette.io/ - because it tries to install pysqlite3-binary which doesn't have a 3.11 release yet:
https://github.com/simonw/datasette/blob/2e9751672d4fe329b3c359d5b7b1992283185820/.github/workflows/deploy-latest.yml#L77
I started using that for the latest.datasette.io demo in https://github.com/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267 because I wanted a version of SQLite that supported generated columns.
Those were added in SQLite 3.31.0 - and the SQLite version in the new base image is 3.34.1 - so I don't actually need pysqlite3-binary any more.

https://latest.datasette.io/-/versions now shows 3.11.

Opened an issue here:

https://github.com/coleifer/pysqlite3/issues/43


[feature] Pose / keypoint detection task

Many of the latest YOLO models (7 and 8) have a different head for the pose task (keypoint detection). It would be amazing if also this project implements it

Hi,
We plan to support this feature after we complete the implementation of the main functions, such as Distributed data parallel and supporting the YOLOv7 object detection model.
Thanks again for your input, and stay tuned for updates!
Best regards,
Hao-Tang, Tsui


Add OIDC nonce support

Recommit https://github.com/firebase/firebase-ios-sdk/pull/4051 (removed sample app part)


Downloaded Brackets, but now I can't open the program.

I tried using FreeViewer, but that download is failing, does anyone know of a free program I could use to open Brackets?  Thanks!

If you downloaded from github (https://github.com/adobe/brackets) then follow directions in README for building Brackets locally. This is necessary if you want to contribute to Brackets core code. If you just want to run Brackets, then download aninstaller from http://brackets.io/.

@kpeacock24 Were you able to solve?

If you still have the problem, please add more details and then reopen.


Problem running docker-backup when docker daemon runs in host mode

Hi,
I'm currently trying to run a volume backup with docker-backup on my docker host. The docker daemon runs in host mode meaning it accepts remote connections on a custom tcp port (-H tcp://<IP_ADDRESS>:30000) but the standard option via unix file is also available (-H unix:// ).
When i try to run the docker-backup with the the following options
docker-backup --addr localhost:30000 --proto tcp store /path/to/my/archive.tar.gz mycontainer

or
docker-backup --addr /var/run/docker.sock --proto unix store /path/to/my/archive.tar.gz mycontainer

i get the error message
Storing mycontainer's volume container as /path/to/my/archive.tar.gz
Couldn't find data container

Am i missing something? Any help would be appreciated.
Thanks
Michael

This tool assumes that you have a dedicated 'volume container' you use with --volumes-from flag, which was best practice back in the days.
I probably should archive this project since I'm not using it anymore. In context of something like Kubernetes you should use volume snapshots etc for backups anyways.


embedded-mongodb does not work with mongo image 6

embedded-mongodb does not work with mongo image 6. MongodbStatusCheck fails, because "mongo" command was replaced with "mongosh"
https://www.mongodb.com/docs/manual/release-notes/6.0-compatibility/#legacy-mongo-shell-removed

fixed in #1156

Already released in 3.0.0-RC2


BrainJS in Node-Red

Hi guys,
Nice meeting you all here.
Has anyone here tried to use BrainJS in Node - Red flow ? Can you guys share any example how to do this using node red.
My objective are :

To create a model to predict :
Input= numerical number range from 50-60, Output= 1
Input= numerical number range from 30-50, Output= 2
Input= numerical number range from 0-30, Output= 3

Use the above prediction model, to predict a realtime mqtt input data

Your help are much appreciated.
Thanks to everyone
Regards,
SD

I already downloaded this to my Node-red palete https://flows.nodered.org/node/node-red-contrib-neuralnet
Can anyone share ,how to create a NN model using this palete.?
Your help are much appreciated.
Thanks to everyone
Regards,
SD

Welcome to our project!
I don't think we have examples of using brain.js with node-red. But we will surely like to help you achieve your goal.
The package you have mentioned is based on much older version of brian.js. I, myself, will try to understand the requirements of the node-red and update the package as necessary.
Please be patient.

Hi there, please try this package: https://github.com/mubaidr/node-red-contrib-brain-js
(This is fork of the above mentioned package, all credits to the original author)

Hi @mubaidr ,
Thanks for your support. I manage to integrate Node-Red and Brain JS.
This is how i do :

Download Brain JS file at here  https://www.npmjs.com/package/brainjs
Save the Brain JS folder inside the Node-Red folder
Edit Node-Red setting.js file to include Brain JS library and save :

functionGlobalContext: {
        // os:require('os'),
           brainjs:require('brain.js'),

Go to Node-Red, design the flow using the function node. And function is like this :

var brain=context.global.get('brainjs');
var fs=context.global.get('fs');
const net = new brain.recurrent.LSTM({
  inputSize: 1,
  hiddenLayers: [20, 5, 1],
  outputSize: 1
});

const trainingData = [
         [0,"low"],
 [2,"low"],
 [4,"low"],
 [12,"medium"],
 [14,"medium"],
 [16,"medium"],
 [22,"high"],
 [24,"high"],
 [30,"high"]

];

const results = net.train(trainingData, { log: true });

var lstmo=net.run([12]);
msg.payload = lstmo;

We must use this function to call Brain Js library.
var brain=context.global.get('brainjs');

Fire the prediction.

I hope this will help.
Thanks


API timeout during deployment

Describe the bug
When deploying our ReportStream application, after the container swap occurs, there's an indefinite period of time (10-30 mins) where the front-end site will timeout after 3 minutes waiting for an API response from only some endpoints.
Impact
Endpoints with this affect, observed 10/18 and 10/20:

Submission History (cleared after 10ish minutes)
Data result size directly affects whether it times out or not: Submissions call with no data returned took just over 2 minutes, Submissions call with data timed out
Organizations endpoint seems fine
Daily Data is slow, but did not time out
No Admin APIs timed out

Expected behavior
After a slot swap, there should be no downtime on the front-end website when calling the API.
Screenshots

Logs
TBD
Additional context

Problem seems to occur on heavier loads BUT all endpoints are slower after deployment
Previously observed on Submission History list endpoint

I will be on Experience deployment duty this week: I will observe what endpoints I can trigger the bug on and list them in this ticket's description.

FYI @GilmoreA6 :)

@jeremy-page @jimduff-usds After observing this last week and paying extra special attention to emerging patterns, my hypothesis is that the container may be under a heavy load at start-up, and paired with the slow performance of the Submission History endpoint, it creates these timeouts. In one instance I was able to query an empty submission history array, but then it timed out on a response that would've had data.
The impact for a theoretical user (not sure how many use the Submission History page) is about 5-10 minutes of downtime every Tuesday/Thursday at 1:30-ish ET. On a rare occasion it's been up to 30 mins.

@jimduff-usds Yes, this is the query as it goes out from our front-end call in prod:
https://prime.cdc.gov/api/waters/org/simple_report/submissions?pagesize=61&cursor=3000-01-01T00:00:00.000Z&endcursor=2000-01-01T00:00:00.000Z&sort=DESC&showfailed=false

Issue removed from Experience board since discovery of impact is done. I'll remain subscribed to track progress!


Move Session model to use the NetBox core ASN model

ported @nahun's commit to latest k01ek/netbox-bgp@develop
Credits: Nathan Wheeler<EMAIL_ADDRESS>https://github.com/nahun/netbox-bgp/commit/6b870b086d46f58cb42e603b5daab3f6fb9d9ee1
NetBox v3.1 added a native ASN model. The plugin should utilize that model instead of a separate model that is specific to BGP.
Fixes https://github.com/k01ek/netbox-bgp/issues/82

Thanks @christianharendt and @nahun for your work, in next release I provide tool for migration plugin ASN objects to native NetBox ASN objects.  So plugin ASN model will be removed later, I think in next minor release.

Thanks @k01ek
I was also considering to create a native ASN object for each plugin ASN during migration, but the native ASN model has a mandatory field: RIR. I was unsure what to insert there as a safe default. Maybe the first RIR found or create a meta RIR "netbox-bgp"?

@christianharendt yea, just create some default RIR. I think change RIR for ASN is easier than  create all ASN numbers by hands.

I've updated the code to be compatible with the latest develop commits