Merge all packages in a single pypi package (for real)

Having tens of pypi packages that are kind of united but not really makes it very difficult to package it in distros, please fix it (also because due to how python includes works there is no real advantage in doing a huge amount of small packages)

Hi @Fale
Thanks for the feedback, but we do not plan to merge all packages into one.
Each endpoint is managed by different service team that didn't have the same deadlines. For instance, it's not because the Batch team made a breaking change, that it's worth updating a big package including Compute and Resources. Actually, most of our users are using 2 or 3 packages (like Compute+Network) and don't want to install the whole services on their machine (size on disk + time to install).
In addition, this is a consistent experience across language.
We are already working with Debian to provide one "azure" package with a frozen state of several packages and we're happy so far with that solution. We will update in a near future the "azure" meta-package, so you'll be able to install one package with one command if you want, but giving the opportunity to other people to precisely chose what they want.

@lmazuel thanks for the answer. Debian is not the only distro out there. I'm trying to understand if the Debian approach is feasable for us as well or not

+1 for Fale's issue
The code base is more like an SDK of individual SDK(s) rather than a single SDK with many classes?

Thanks @CalvinHartwell for your comment :)
@Fale I agree that the cross language point is not interesting, let's forget I said that :)
I cited Debian as an example because it's recent work for us, but I know there is more distro, RHEL, CentOS, Suse, ArchLinux etc (Mandriva when I was young...). Please be sure I keep an open-mind here and I'm very interested in the conversation (I really do).
What I don't get is why you think a meta-package "azure" that installs some other package is a problem. Meta-package is something common in the Linux world, It's the whole point of the dependency system. In this situation, you have (for instance with random version number) a package "azure" 2.4.2 that will install "azure-mgmt-resource" 0.45.0 and "azure-mgmt-compute" 0.65.0 at the same time. So, you apt-get/yum install azure and got several Python packages at once. What's the issue?
Currently, the "azure" meta-package is not accurate because some core libraries are still in preview. But the final plan is to make sure that "azure" will install every stable packages at the same time, with fixed version. "preview" services will be available for testing as separate package if you want.
Please note that we also have people that are happy to install exactly what they want (only the Azure services they need).
Thoughts?

The current implementation has 3 problems from my point of view:

You put a huge amount packages in the same repo. I get that this approach has some advantages, but is exactly the opposite of git and python best practices and common practices
Every single package has it's own release cycle (and therefore version number), so the azure one, is not a real meta package but an installer.
If this is an SDK, should have a single release cycle and release number, otherwise is a collection of libraries (which is not the same thing as an SDK)

Also, the point of "many libraries" is a problem during the packaging effort because for many distro (included Fedora, and *EL, for which I'm looking at this SDK) every single pypi package has to be managed in a different (rpm) package, and therefore this means that to maintain the Azure SDK and Azure CLI I will have to maintain ~50 packages. On the other side, I'm the mainter of the AWS python SDK and AWS CLI tool and those are only 3 packages (2 for the SDK, 1 for CLI).

@Fale To discuss, let's assume I merge everything into one package. Botocore is using meta-descriptions of RestAPI "on the fly", we use our meta-descriptions to generate Python code, which takes more place on disk. Our biggest package (Web) is currently doing 4Mb. On average package are 3Mb sized. This means that 40 services of Azure, we can estimate the azure package to reach 120Mb. We plan to support several APIVersion for compatibility in a near future, which can leads a Python package of 500Mb easily. Of course it's an estimate, but it's a likely scenario. What do you think?

@Fale Also, why can't you put inside one rpm file several python packages at the same time? Is there some technical limitation somewhere, or is it just non conventional?

Not be be read in a bad way, but your point is that your code is bloated and for this reason is better to split it? This argument is not very strong, I think...
About the fact of putting more python packages in the same rpm is against the policy in many cases, also in the specific case is not possible (technically speaking) due to the fact that a pa page has a single version and all azure packages have different versions and using wrong version is definitely against the policies

If you have a strong equivalence azure 2.1.2 == azure-mgmt-resource 0.40.0 + azure-mgmt-compute 0.35.0 and so on, why don't you create a python3-azure package 2.1.2? There is no ambiguity, no cheating, and you have a single version to use. It's the approach used by Debian, and I don't see where it's against any policy? Really, I'd like to understand your point, but I'm not seeing the technical problem yet :(

@rjschwei @bear454 @schaefi, would you like to share your point of view for Suse?
@irl, would you like to share your point of view for Debian?

Fedora guidelines force us to package things as much as possible "as the upstream" does.
So, currently I should do the following packages:

python?-azure 2.1.2
python?-azure-mgmt-resource 0.40.0
python?-azure-mgmt-compute 0.35.0
etc

Can I create the package python?-azure 2.1.2 that ships all files of the other modules? No
Why?

It violates the "stick to what upstream does" policy
It will become a mess to manage due to the fact that other packages that will depend on specific libs of the SDK (probably the majority) will have dependencies to that specific module and to a set of versions (or a specific version) which will not be 2.1.2.

I can do what you describe only if the release cycle and the version number will be the same for all modules.

So originally we also struggled with the issue of many Python packages vs. one rpm package which is why we opened the other issue and it took us a while to wrap our head around how to approach this. We finally decided to basically follow the Python package strategy.
So what we have today are 2 packages https://build.opensuse.org/project/show/Cloud:Tools?search=azure, python-azure-sdk and python-azure-sdk-storage. As the whole thing gets broken into smaller pieces with different upstream teams managing different code streams I see the argument about the coordinated release problem.
Also note that we decided to pull our sources  from GitHub rather than pypi. We did struggle with the way things are pushed to pypi and found pulling from GitHub to be a better approach for us for package creation.
To a certain degree we/I followed a similar approach with the ec2utils we provide in our Enceladus project, https://github.com/SUSE/Enceladus/tree/master/ec2utils, meaning different release cycles for each utility.
Having different Python packages, which for us will eventually translate into different rpms implies that client code, for us azurectl, https://github.com/SUSE/azurectl, can be more precise about dependencies, which is an advantage. We have not yet packaged the new az tools, thus I cannot speak to the effect on that packaging effort and dependency management from that point of view, but I am certain we'll solve that in a reasonable way.
While I share the concern of @Fale regarding package proliferation as well as the multi Python package approach not being "Pythonic", there are equally valid arguments on the other side, meaning to have a sdk-storage package, maybe one for networking etc. I think the argument about being "Pythonic" mostly comes into play after install. Meaning as long as I can
from azure.storage import ....
from azure.networkig import ...
as a Python developer I really do not care whether site-package/azure/storage and sit-packages/azure/networking were installed by 2 distro packages or 1.  Having multiple packages may be a bit more cumbersome for the developer to set up the system, meaning the developer has to potentially install many packages but that can be easily done with a one liner:
pckgmgr --search python-azure | grep sdk | xargs pckmgr install
or a meta-package, i.e. we can easily create a python-azure-sdk-all package or create a pattern that pulls all the other packages.
I think it is equally valid to look at each service Azure provides as a separate target and have a separate SDK for that target as it is to look at the API as a whole.
That it was decided at  the origination of boto, and now carried into botocore, that all of the AWS API should be in one SDK, one Python package, is just as valid a decision as the decision made here that each service should have its own SDK managed by separate teams.
So long story short, from my perspective either way is fine. If from a development perspective things are managed more easily at Msft to have multiple Python packages we are game to follow that route.

Referencing related issue created for Python CLI which also has separate packages - https://github.com/Azure/azure-cli/issues/1055

From the Debian perspective, I am ignoring PyPI and packaging from the Git repo. The idea that in the future there will be changes to the individual packages which will then be released via PyPI and not git tags breaks this. If the releases of all the core modules (i.e. the ones in this repo) are synchronised it makes everything a lot easier for me, and I guess for other distros too.

Thanks @rjschwei and @irl !
So, in summary, what I understand is as long I create some checkpoint as tags in the repo, you're good to package the current Github state as "tag" version number. I can publish on PyPI new packages for a specific service if available, but it will be sync as a Linux package only when I release a new version of the "azure" meta-package (with a new associated tag on Github).
That's seems fair to me. Anyway, I plan to release more often the "azure" meta-package once the core ARM modules (Storage/Compute/Network/Resource) will be officially stable.
@Fale are we fine with that plan?

It will be long and painful, but I can make it work with the policies. Thanks

So, in summary, what I understand is as long I create some checkpoint
as tags in the repo, you're good to package the current Github state as
"tag" version number.

yes this makes packaging work much easier. The same code base referencing
that release tag should also exist on pypi. In my projects that happens
automatically see:
https://docs.travis-ci.com/user/deployment/pypi

I can publish on PyPI new packages for a specific
service if available, but it will be sync as a Linux package only when
I release a new version of the "azure" meta-package (with a new
associated tag on Github).

yes and it should be possible to make this an automatic step
Regards,
Marcus

@Fale fwiw - "do what upstream does" doesn't have to mean PyPI as PyPI is a downstream distribution of the upstream, there's no reason not to package up the sources from Git (possibly my Debian-oriented frame of reference). I have one source package, but plan to build one binary package for each of the logical PyPI packages within that.
@lmazuel That's perfect for me (:
@derekbekoe This would work great for the azure-cli package also.

@irl: I think I'll go the github way too. We do this for many situations :). I'll go for one src.rpm and multiple rpms, but even having a single src.rpm package, it will be a fairly complex spec to be able to generate all the various sub packages properly considering files and versions etc

@Fale I will just be shipping everything with the version number of the metapackage. Subcomponents may have differing versions, but they're all part of the larger "unified" release.

@irl How do you manage packages dependencies that depends on the subcomponents? ie: https://github.com/Azure/azure-cli/blob/master/src/azure-cli-core/setup.py#L49

@Fale If there's a tagged unified release in git and the dependencies don't line up, then Microsoft has done a terrible job at release management. I don't anticipate this happening often.

@irl Microsoft point is that they want to have different version numbers to be able to have different development cycles for the various parts of the code, so I anticipate this happening often going forward

@Fale yes, between releases it may be broken and not all lined up, but the metapackage needs to be released with everything lined up otherwise it will never be installable, so the idea would be to package in distributions when, and only when, the metapackage sees a release and the git repo is tagged.

Also:

people could "cherry pick" an update and break the other
other packages (non-Microsoft) could depend on Microsoft code

@Fale not through the package management system they can't, and it's not a bug in your system if they've done something to break it. I fully anticipate other packages depending on the sdk, I have vagrant-azure in Debian depending on the Ruby SDK, and I'm quite happy to continue supporting this. I've had to patch the crap out of it to get it to work with the latest SDK, but as a distribution packager I expect to have to do some work occasionally.
From what I can see, Microsoft are new to this and I'd rather compromise and accept a little extra work than make demands that they change their entire project management workflow and put them off engaging in open source communities. This situation is no different from any other situation where you have a library and it has dependencies, some external. As a distribution packager, you should be performing QA to catch these problems and working with upstream to find resolutions, or patching locally within your distribution to ensure all your packages line up.

A couple of points and then I'll stop with this since we are going OT:

Microsoft are new to this and I'd rather compromise and accept a little extra work than make demands that they change their entire project management workflow and put them off engaging in open source communities.

Since Microsoft is new, is even more important to discuss with them "how open source works", so that they can understand it before doing errors
I can compromise and accept extra work, I will not compromise and accept to break Fedora dependencies

@Fale I think we're aggressively agreeing with each other perhaps. The important thing is that there are releases of the metapackage that have all the dependencies working together nicely.
To summarise my view:

Releases get tagged in the git repo
When a release is tagged, all the modules will work together
Distributions will update packages when a metapackage is released
It is expected that changes in modules will be backwards compatible to the last version released with a metapackage, or the metapackge version gets bumped (version numbers are cheap)
It is expected that changes will not be massive and sweeping every release allowing for others to build applications against the SDK
Distribution packages are aimed at production use by end-users, developers are still able to install more recent releases for PyPI in order to test their applications with future versions
It is expected that if a developer breaks their system by mixing versions from distribution packages with PyPI, they should be able to understand why it is broken

There is a great article on the topic of packages vs. pip here: https://notes.pault.ag/debian-python/

Thank you for your contributions, it really matters :). I agree with latest @irl comment.
In full transparency, the next possible breaking change might be for Azure Stack support in the SDK in a few months. I will bump a major version for every packages at the same time, and keep it in a parallel branch in preview as long I'm not sure than Ansible and others folks I'm in contact with are not ready (FYI I wrote the Ansible plugin with a workmate at MS, I really care to make it work). We don't plan a major change, but the way we have to create the client and authenticate against Azure might change enough to justify a version bump. For now it's subtle enough, that we can provide one line at the start (like Python2/3 compat) to support both versions at the same time. We'll try to keep it that way.
Do not hesitate to contact me by Github or direct email<EMAIL_ADDRESS>if you have any questions or concerns.

I was trying to package this repo as suggested by @irl  and @rjschwei and I noticed a problem with a circular dependency with this approach:
azure (aka this repo) depends on azure-storage which depends on azure-common (aka this repo).
How have Debian and Suse manage to make this work?

In Debian, we have an unstable distribution where we can have things that have broken dependencies. In practice, building python-azure (as we call this repo) does not depend on python-azure-storage, only running it. We can upload python-azure and python-azure-storage and our tools will automatically move these to the testing distribution (currently stretch) when all the dependencies line up.

You can see our spec file here:
https://build.opensuse.org/package/view_file/Cloud:Tools/python-azure-sdk/python-azure-sdk.spec?expand=1
and as mentioned we do have a separate package for storage, the specfile is here:
https://build.opensuse.org/package/view_file/Cloud:Tools/python-azure-sdk-storage/python-azure-sdk-storage.spec?expand=1

@irl thanks
@rjschwei do you have some king of "auto-generating" dependencies? Because I don't see any dependency declaration to the other package on those two spec files

[2]@rjschwei do you have some king of "auto-generating" dependencies?
Because I don't see any dependency declaration to the other package on
those two spec files

the azure storage api does not depend on the azure service management api.
Thus python-azure-sdk and python-azure-sdk-storage do not have a dependency
on each other. Our python-azurectl utility however depends on a specific
min. version of the storage and servicemanagement api in order to work
correctly with the sdk features azurectl uses

Back in the day, we (the Python team) made the first azure-storage package, so it was on this repo. When it was obvious that we had some performance issue and that storage drove a lot of specific questions, the storage team took ownership directly to improve it drastically and be more responsive. This implies that the "azure-storage" package is close to our package structure and use our meta-package, but is in another repo (unlike DocDB for instance).
I didn't realize that this could be a problem here :(. It seems to me that azure-storage should be in the Linux package "as if it were on the azure-sdk-for-python" repo. This solve the dependency problem, but I agree it makes the situation a little more complicated :(.
It's the only exception for history reason. We do not plan to include more data stuff not in this repo, like DocDB, in the azure meta-package. We are working on azure-keyvault, but this will be in this repo as well.

@schaefi it seems like azure-sdk-storage depends on azure-common (https://github.com/Azure/azure-storage-python/blob/master/setup.py#L68) which is in this repo.
@lmazuel the majority of distribution do not accept multiple git repos as source of a single package, so the azure-storage can not be provided by the same source package of the python-sdk. As @irl suggested, it would be possible to build both packages and then push them together to stable, but this would mean that no tests can be performed during the build since at build time you can not install the other package (since it would depend on the package you are building at the moment) and for Fedora tests should be performed in build time (it's not mandatory but it's highly suggested). Also the "push together" approach could have other problems during the life of the package.

Ok, so I guess the best approach is to leave azure-storage in it's own package. azure-storage needs azure, but we can cut the opposite link (azure does not really need azure-storage, no code is using it in any package). This avoid you circular link.
I think in the long-term the dependency of azure-storage on azure-common will disappear. azure-common is used only for legacy code now in this repository. This will simplify the situation.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Mon, 2016-10-17 at 09:45 -0700, Laurent Mazuel wrote:

I think in the long-term the dependency of azure-storage on azure-
common will disappear. azure-common is used only for legacy code now
in this repository.

Is "legacy code" here used in the generic way, or are you specifically
referring to ASM ?

James Mason
Technical Architect, Public Cloud
openSUSE Member
SUSE
<EMAIL_ADDRESS>-----BEGIN PGP SIGNATURE-----
iQEcBAEBCAAGBQJYBSCxAAoJEBs5UYhsRJAjxHIH/i9sKqGEE1r5G8n1tdl8FKqj
R1/btmovLYRKst1tUXHw2pbEgeiUK1K5n+QAid1nZ42lYb+E9/XZLqnj0shuZthM
H+I8B8UojgV8bpDczYvQKFzR0XK8O35fbEbzIi3O/P2g5ehqI7vadh2ObeIRuGS9
jchBpT2qSqPnih04dKgzde+3DvO2ePq58y2rX/uud4lyNLuZTPPwHjlV7/62k9YR
uLwnX1S1XZnpsZyL5V+Xjz4a7Kxp5bcdR5drl2zs5KgPwo7dHK/XozSJLodq/8Gm
dDO7pGtJNOapN74YK1qjwzQnsSVT2TZle8ph8YhHzmU6blIMuxF3h4LqIIm3Tjk=
=SQxG
-----END PGP SIGNATURE-----

@lmazuel the dependency dropping would solve this circularity problem :)

@bear454 I mean "no ARM", so ASM + azure-servicebus.
All common libraries for ARM are in msrest/msrestazure packages.

Is there a chance we could at least see another bundled release? I'm trying to maintain packages for azure-cli on Arch but I've run into a problem where the current 2.0.0rc6 release is too old and results in module errors and the git builds are too new resulting in a different set of errors.
I've tried building each module in this repo separately but there's a ridiculous amount of them and several of them seem to be unable to install independently as needed for Arch packaging.
I can see the argument for this because of independent release cycles but as a user I don't care. I need something that just works. The easiest solution, I think, would be to package major releases every so often that contain stable versions of each module.

Hi @optlink
Yes, the rc7 is planned. The problem is still that for a meta-package to be tagged as "stable", I need all sub-dependencies as stable. And it's not the case currently.
However, if you do packages for the CLI, the CLI uses the same behavior: they are cut into services and sub-packages . For instance, this is the Network implementation of the CLI. And this package follow directly a specific version of Network (this 2.0.1 is linked to 0.30.0). So even if I do a rc7, I can't assure that at all packages will match all the sub-dependencies of the CLI. And even if I assure it today, this can change tomorrow with an update of Network or something else.
I'm not sure I understand clearly your constraints (I'm a Ubuntu user, I just know Arch by name sorry :-( ), but send me an email at MS (<githubalias> at microsoft.com) and we will discuss with the CLI team how to help, or at least brainstorm something.
FYI @derekbekoe @johanste

On 03/30/2017 11:03 AM, Kelsey Maes wrote:

Is there a chance we could at least see another bundled release? I'm
trying to maintain packages for azure-cli on Arch but I've run into a
problem where the current 2.0.0rc6 release is too old and results in
module errors and the git builds are too new resulting in a different
set of errors.
I've tried building each module in this repo separately but there's a
ridiculous amount of them and several of them seem to be unable to
install independently as needed for Arch packaging.
I can see the argument for this because of independent release cycles
but as a user I don't care. I need something that just works. The
easiest solution, I think, would be to package major releases every so
often that contain stable versions of each module.

For what it's worth. So far we have also created a bundled package for
openSUSE and SUSE Linux Enterprise. However, we are starting with
packaging the az tools (azure-cli) and it is also split into many
pieces. Those in turn depend on the pieces of the SDK rather than the
SDK as a whole. Thus as a package you end up having to either maintain a
large number of packages or a large number of directives for provides in
order to sort out the proper version dependencies. We are going down the
road of many packages.

Thank you @rjschwei for your message. We will still continue to release one package per service, but do you have a suggestion of zip/tar.gz/package/tools or something that we can do to simplify your process?

On 04/03/2017 12:19 PM, Laurent Mazuel wrote:

Thank you @rjschwei https://github.com/rjschwei for your message. We
will still continue to release one package per service, but do you have
a suggestion of zip/tar.gz/package/tools or something that we can do to
simplify your process?

I don't think there is anything else to do. The only simplification
would be to release everything as one, but we've already had that
discussion, and then have the cli package depend on that SDK version.
Anyway, since the cli gets released as Python packages based on service
components that in turn depend on Python packages that are released per
service component that's is really the model we have to follow on the
packaging level.

Hi!
I have recently picked up the task to package the Azure SDK in openSUSE. For openSUSE, the current plan is to use the packages from the PyPi repository. However, while working through the various azure-mgmt-* packages I noticed that many packages are either outdated on PyPi ( those are commerce, compute, network, powerbiembedded, resource, servicebus and storage from the mgmt packages and the meta packages azure-mgmt and azure-nspkg) or are missing the __init__.py files so that setuptools fails to install them properly (those are eventhub, media, network, resource, search, servermanager, servicebus and storage).
On the other hand, I could also use the tarballs generated by the git tags in the github repository. However, it's not clear to me which of these tags should be used when packaging the whole SDK while ensuring all modules are compatible with each other. If I read the discussion correctly, some of the modules can be too new so that they won't work with certain other modules anymore and can only be used individually. And if one wants to deploy the whole SDK, all modules must have the version belonging to a particular version of the whole SDK.
So, my question now is: How do I get release tarballs with the proper versions for each module so that I get a complete and working SDK in the end? PyPi is currently apparently not the best source for the aforementioned reasons and so are the released creates through the git tags on github.
Thanks!

I just figured out that the SDK releases are available as single tarballs generated from the git tags, they follow this pattern:
https://github.com/Azure/azure-sdk-for-python/archive/v?(\d.*)[A-Z,a-z,0-9]*\.zip
e.g.:
https://github.com/Azure/azure-sdk-for-python/archive/v2.0.0rc6.tar.gz
So, I suggest just pulling the tarball from there and using this as a base for the packaging.

Hi,
On 05/10/2017 05:02 AM, John Paul Adrian Glaubitz wrote:

I just figured out that the SDK releases are available as single
tarballs generated from the git tags, they follow this pattern:
|https://github.com/Azure/azure-sdk-for-python/archive/v?(\d.*)[A-Z,a-z,0-9]*\.zip|
e.g.:
|https://github.com/Azure/azure-sdk-for-python/archive/v2.0.0rc6.tar.gz|
So, I suggest just pulling the tarball from there and using this as a
base for the packaging.

That doesn't work because the azure-cli releases depend on the
individual components of the SDK, not on the SDK as a whole.
So as packager there are two choices:
a.) Create 1 package for SDK, as we pretty much do in openSUSE right now
and then have a very long list of Provides: statements where each
Provides lists a component. This list is going to be a PITA to maintain
and will inevitably be wrong and cause headachs
b.) Package each individual component of the SDK, the approach we are
now taking.

@derekbekoe, do you have any suggestions?

Hi @glaubitz, sorry I didn't answer earlier, it was busy with //build/ this week and PyCon next, and I wanted to take the time to answer you correctly. I just want you to be sure I don't ignore you, I'll be back with my full brain :)

That doesn't work because the azure-cli releases depend on the individual components of the SDK, not on the SDK as a whole.

Sorry, I wasn't clear enough then. I was not talking about creating a single RPM package, but to use the github tarball as a single source. Not because I particularly prefer github over PyPi but rather because the packages on PyPi are either outdated or broken.

a.) Create 1 package for SDK, as we pretty much do in openSUSE right now and then have a very long list of Provides: statements where each Provides lists a component. This list is going to be a PITA to maintain and will inevitably be wrong and cause headachs

I agree and that's definitely not what I want. However, having to pull every

b.) Package each individual component of the SDK, the approach we are now taking.

That's definitely what I want to do. However, my problem currently is that I don't know for sure which set of packages I should use.
Should I:
a) Use the v2.0.0rc6.tar.gz as the source for all base packages and create RPMs from that? I have written a small script which creates the individual .zip files for all individual packages. Then complement these RPMs packages with the remaining packages from our list, just using the latest available release version for each package.
or
b) Just use the latest tarball available in the github "Releases" tab, unpack that archive and generate the individual .zip files from there? For example, downloading https://github.com/Azure/azure-sdk-for-python/archive/azure-keyvault_0.3.3.tar.gz, unpacking it and creating the individual .zip files using that archive.
The reason I ask is because each of these tarballs always contain the complete SDK and not just azure-keyvault, for example. Thus, when I download and use the tarball azure-keyvault_0.3.3.tar.gz, can I still assemble a working SDK from that or does that work with the v2.00rc6.tar.gz tarball as it has been tagged as a release of the whole SDK?
It's just confusing that the individual packages and the complete SDK show up on the same "Releases" tab. A releases normally indicates something that is stable - or at least beta - that users can download and use.
Adrian

To elaborate a little more: I just ran my small script over the unpacked azure-keyvault_0.3.3.tar.gz and it created azure-2.0.0rc7.zip among others, so the resulting SDK I got is something between rc6 and rc7 (since rc7 has not been officially tagged yet).

@glaubitz I'm not really sure that managing tens of highly coupled packages is anywhere easier.... and that's why I opened this ticket (aka: I think it is not possible to manage this project in a sensible way, and this is why - given the IMHO unsatisfactory answers - I'm not packaging this for Fedora/EL)

@Fale But you can just download the tarballs from the github releases page and you get all modules in a single tarball. In fact, that's what @irl is doing for Debian and since the Debian version is currently at v2.0.0rc6, it has less modules than are currently visible in the github repository.
I generally don't have a problem juggling with a large number of sources - it's just a matter of good packaging tools after all - I'm just confused as to which versions to use for a stable distribution.

Hi @glaubitz
I understand it's complicated, really, :/. Github is not really built to host several packages in one repo. This is some answers:

Tag are on purpose "<package>_<version>" and are made just for the specific package mentioned in the tag. I don't recommend for instance to use tag "azure-keyvault_0.3.3" to install "azure-mgmt-compute"
Tag like "v2.0.0rc6" are also intended to be available for "azure 2.0.0rc6" only, even if I'm pretty sure the state of the repo at this state was correct, according to the content of v2.0.0rc6
Package on PyPI are not outdated, I'm surprised you got issues? About the issues you found, could send me a more detail email at<EMAIL_ADDRESS>
Let's be pragmatic on what you want (before talking about how to do it): do you want to release one package like Debian like python-azure 2.0.0rc6? Or separate packages for each components? As @rjschwei was saying, the CLI is using each component package independently, so we might have an issue with that. Let's say we can sync azure-cli and azure-sdk bundle package, do you want to:

Release python-azure x.y.x with a lot of packages
Release python-azure-cli x.y.z that depends of python-azure x.y.x?

Once I get what you want the user experience to be, we will figure out the "how".

Hi @lmazuel

I understand it's complicated, really, :/. Github is not really built to host several packages in one repo.

You could put each package into a separate git repository and then use git submodules to references these modules in the git repository for the whole package. Lots of projects actually do this when they use third-party libraries like ffmpeg.

Tag are on purpose "_" and are made just for the specific package mentioned in the tag. I don't recommend for instance to use tag "azure-keyvault_0.3.3" to install "azure-mgmt-compute"

Ok, so this means that despite azure-keyvault_0.3.3 containing the whole SDK, I should just always assume the remaining packages are effectively git snapshots and should not be used for anything but development. Thus, when I download azure-keyvault_0.3.3, the azure-mgmt-comput package inside this tarball is probably version 1.0.0rc1 plus some extra commits and shouldn't be used for production.
Thus, anyone wanting to use the releases from github really needs to download the tarball separately for each tagged package version.

Tag like "v2.0.0rc6" are also intended to be accurate for "azure 2.0.0rc6" only, even if I'm pretty sure the state of the repo at this state was correct, according to the content of v2.0.0rc6

Isn't azure supposed to be the primary meta package which allows to install the whole SDK in one step? I'm not sure what would be the point of tagging a version of number for the whole SDK if it doesn't mean the generated tarball doesn't create something that works.

Package on PyPI are not outdated, I'm surprised you got issues? About the issues you found, could send me a more detail email at @microsoft.com?

Sure. Will do that once I have finished writing this message ;).

Let's be pragmatic on what you want (before talking about how to do it): do you want to release one package like Debian like python-azure 2.0.0rc6? Or separate packages for each components?

I want to release separate components. But I also want that these components work with each other, at least that's what users are going to expect. If they use the package manager to install azure, they expect to get the SDK installed ready to be used without having to replace individual components.
For me as the packager, it doesn't really matter whether the whole SDK is released in one tarball or as individual packages. I am writing some simple scripts that will help me deal with the upstream format to generate the RPM packages. What matters is that I know which versions I have to use to be able to assemble something that is going to work in the end on the users side.
For example, if you have released any of the packages in a version which breaks compatibility with most of the other packages, I will naturally not use the latest version of that particular package. I will use the version which is still compatible with the rest and only update once all the other packages have made the transition upstream.

As @rjschwei was saying, the CLI is using each component package independently, so we might have an issue with that. Let's say we can sync azure-cli and azure-sdk bundle package, do you want to:
Release python-azure x.y.x with a lot of packages

Yes, that's what I want. But again, creating a single package out of individual packages or vice versa is not the actual problem. The problem I  have is that I don't know which versions are compatible with each other to form a complete, working SDK.

Once I get what you want the user experience to be, we will figure out the "how".

So, here's what I suggest:
If I understand correctly, all the various packages are developed separately. So, these packages should naturally end in separate git repositories. Then use git submodules to link the packages in the main git repository of the Azure SDK. git submodules allows to link specific git commit versions of another repository. Thus, you are able to assemble the SDK from specific versions that are known to work together and you always have something releasable.
If users want to use individual packages, they'll download the tagged tarball from the corresponding package's repository. If they want the whole SDK, they just download the latest tagged version as a tarball.

@lmazuel
Ideally we'd have 1 upstream tarball for each, the SDK and the CLI such that we can create
python-azure-sdk-x.y.z and azure-cli.a.b.c packages with azure-cli.a.b.c depending on python-azure-sdk-x.y.z
That's how the other guys do it ;) aws-cli has only a few dependencies with python-botocore being the equivalent to azure-sdk as the primary dependency.
Anyway, I understand, as does probably everyone else interested in this topic, that there are tradeoffs either way and going with a development model of individual components is just as valid a choice as going with a development model that keeps everything together. However, with the chosen model of many components people down stream (packagers or direct users) still need to have some moment in time every now and then where all the pieces fit together. Based on the finding of @glaubitz this point in time is incredibly difficult to determine.
So somehow a mechanism should exist that allows us to pull what would be considered a consistent SDK. If the answer to that is "whatever is on pypi" then that's OK, and maybe we just have to clean up a few things that @glaubitz ran across on pypi and then we are good to go.

@glaubitz @rjschwei
About SDK consistency:

For packages who depends on msrestazure, they must be have ">= 0.4". This is the only condition, meaning you can install azure-mgmt-resource 0.30.0rc6 and azure-mgmt-compute 1.0.0rc2 together with no issue. It's consistent in terms of installation, it's just weird in terms of features.
For packages that not depends on msrestazure (I think there is three only, azure-servicebus, azure-servicemanagement-legacy and azure-storage), they are independant and consistent from version 0.20.0

Also, the source code truth is the sdist on PyPI. It's easy to get with XMLRPC, example for azure-keyvault 0.3.3:
import xmlrpc.client
client = xmlrpc.client.ServerProxy("https://pypi.python.org/pypi")
[pkg['url'] for pkg in client.release_urls('azure-keyvault', '0.3.3') if pkg['python_version']=='source'][0]

gives
https://pypi.python.org/packages/82/8b/9761cf4a00d9a9bdaf58507f21fce6ea5ea13236165afc0a0c19a74ac497/azure-keyvault-0.3.3.zip
I'll discuss it with the CLI team today, I'll see if we can sync our release (for instance each 6 months). I want to release a 2.0.0, and I will try to use the exact same package than CLI 2.0.6. This way you can package azure-python-sdk 2.0.0 as a whole, and package azur-python-cli 2.0.6 as a whole as well, depending of azure-python-sdk 2.0.0
Thoughts?
FYI @johanste

On Mon, May 15, 2017 at 10:44:03AM -0700, Laurent Mazuel wrote:

About SDK consistency:

For packages who depends on msrestazure, they must be have ">= 0.4". This is the only condition, meaning you can install azure-mgmt-resource 0.30.0rc6 and azure-mgmt-compute 1.0.0rc2 together with no issue. It's consistent in terms of installation, it's just weird in terms of features.
For packages that not depends on msrestazure (I think there is three only, azure-servicebus, azure-servicemanagement-legacy and azure-storage), they are independant and consistent from version 0.20.0

Thanks. This answers my question.

Also, the source code truth is the sdist on PyPI. It's easy to get with XMLRPC, example for azure-keyvault 0.3.3:
import xmlrpc.client
client = xmlrpc.client.ServerProxy("https://pypi.python.org/pypi")
[pkg['url'] for pkg in client.release_urls('azure-keyvault', '0.3.3') if pkg['python_version']=='source'][0]

gives
https://pypi.python.org/packages/82/8b/9761cf4a00d9a9bdaf58507f21fce6ea5ea13236165afc0a0c19a74ac497/azure-keyvault-0.3.3.zip

Aha, I wasn't aware of that. Thanks for the heads-up!

I'll discuss it with the CLI team today, I'll see if we can sync our release (for instance each 6 months). I want to release a 2.0.0, and I will try to use the exact same package than CLI 2.0.6. This way you can package azure-python-sdk 2.0.0 as a whole, and package azur-python-cli 2.0.6 as a whole as well, depending of azure-python-sdk 2.0.0
Thoughts?

We wanted to have separate packages in SUSE anyway, so that isn't
important. I really just wanted to know whether the version
dependencies are critical.
Thanks,
Adrian

On 05/15/2017 01:44 PM, Laurent Mazuel wrote:

@glaubitz @rjschwei

> I'll discuss it with the CLI team today, I'll see if we can sync our release (for instance each 6 months). I want to release a 2.0.0, and I will try to use the exact same package than CLI 2.0.6. This way you can package azure-python-sdk 2.0.0 as a whole, and package azur-python-cli 2.0.6 as a whole as well, depending of azure-python-sdk 2.0.0
>
> Thoughts?
>
That would be great but would require significant changes in the setup
of the CLI, i.e. within the components of the CLI the dependencies in
setup.py could no longer refer to the individual components of the SDK.
That's a bunch of work that will probably not fit the development model.

@rjschwei I'm not sure I get your issue? When you install a distrib package like python-azure-sdk, I think you install the necessary "dist-info" folders, so pip is not able to make the difference between a yum installation and an pip installation correct? So here, if your python-azure-cli depends on python-azure-sdk, and I took care to make them in sync, this should on the contrary makes your life easier?
@irl what do you think about that? Because if trying to sync SDK and CLI bundles makes no sense, I have no reason to do it.

@lmazuel , sorry for falling off the face of the planet for a bit and creating a large time gap in the discussion.
You are correct that the installed rpm package will also leave behind the Python information to satisfy installing the CLI bits. Thus if SDK and CLI releases can be synced such the cli-a.b.c depends on sdk-x.y.z then we could go to a one package model and we'd basically have 1 dependency in the CLI package.
However, my concern with this approach would be that, to the best of my knowledge, no tools exist today to ensure this consistency. Of course such tools can be created, but in a sense these tools would counteract the development separation that at this time has been instituted in the SDK and CLI projects.
So if you'd go through the effort to sync everything, which would really be nice for packagers, I think the development model would have to change to a certain degree.
Getting everything in sync would basically mean to collect all the components and verify their dependencies are consistent within each, the SDK and the CLI and are consistent across the boundary. Creating a tool that ensures such consistency should be reasonably straight forward, but it still has to be created and maintained.
However, during the "development phase" this consistency is not necessarily given, meaning CLI component A may depend on version X of SDK component H and CLI component B may depend on version Y of SDK component H. Which is fine as long as at the end of the development cycle both CLI components A and B depend on the same version of SDK component H. This drift makes testing very difficult. Also when there is a security issue because continuous testing is difficult it will not be a good idea to release the security fix from the development branch. The security fix will have to be inserted in two places, the current consistent (synced) code and the development code with a point release off the previous consistent set. This of course can all be managed, but the point is that developers on two teams will have to work more closely together than it appears was intended when the current development model was chosen.
If we look at the same problem using the many-packages approach we can still get into a similar situation, if SDK component H gets a security fix and the version gets advanced. Now the CLI may be broken. However, because the dependencies are not conglomerated we know exactly which CLI packages need potential updates to accommodate the version bump of SDK component H.
To make a long story short, a sync will make initial packaging easier, but individual packages will make dealing with version bumps due to security issues easier.
One thing that would help tremendously would be if you and the CLI team can commit to semantic versioning http://semver.org/ at all levels and change the dependencies in all setup.py files accordingly, i.e.every dependency should be >= one major version and <= the next major version. There should not be any exact version matches enforced. If we can get to that point managing the plethora of packages will be reasonably straight forward.

Closing, in favor of https://github.com/Azure/azure-sdk-for-python/issues/1295


[1.12] [http1] Preserve LWS from the middle of HTTP1 header values that requ…

…ire multiple dispatch calls to process (#10886)
Commit Message:
Correctly preserve linear whitespace in the middle of HTTP1 header values. The fix in 6a95a21 trimmed away both leading and trailing whitespace when accepting header value fragments which can result in inner LWS in header values being stripped away if the LWS lands at the beginning or end of a buffer slice.
Signed-off-by: Antonio Vicente<EMAIL_ADDRESS>Signed-off-by: Yuchen Dai<EMAIL_ADDRESS>Additional Description:
Allow compiling on clang-10
Risk Level:
Testing:
Docs Changes:
Release Notes:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]

/azp run

I will fix the DCO after

I may accidentally introduce the cx_integration_test...
Sorry I need to force push


icons showing up as squares

Not sure why but my icons are not showing.
I have tested some other things and it seems to work, but icons don't.
This is just a sample project.
https://github.com/dmc2015/roku-app
You can do 'npm install' and then 'electron src/index.html'

Hi @dmc2015,
That’s because your app’s photon-entypo font is overriding SUI’s Icons font:
.btn-dropdown:after, .icon:before {
  font-family: photon-entypo;
  // font-family: Icons;
}

The following is a standard reply.
Thank you for posting, but although it’s a valid usage question, we’ve limited GitHub Issues to bug reports and feature requests, keeping the board more manageable for maintainers; see the contributing guidelines for more information on what kind of posts should find themselves into the GitHub Issues board.
To get answers or feedback that might allow you to repost this issue, please use one of our other support resources:

Gitter chatroom
Official forums
StackOverflow


Remove animation of search input

Describe the problem
It makes the website look unprofessional
Describe the proposed solution
Remove animation of it
Alternatives considered
not considering any alternatives
Importance
nice to have

Thanks for your feedback!
This is a very subjective opinion.
Would consider changes if there are some objective data to support your opinion


Install on Arch Linux

Hi,
Following README :

hexyl AUR paskage no longer exists

In official ArchLinux AUR packages
I only find:

hexyl-git AUR package
hexyl-bin AUR packages

README should be corrected and updated?
Thanks

README should be corrected and updated?

Yes, thank you for the feedback. Ideally, both packages would be mentioned, but the default should be hexyl-bin.

The package has been added to the official Arch packages repository, which is why it is not in the AUR anymore!
https://www.archlinux.org/packages/community/x86_64/hexyl/
It can now be installed with pacman -S hexyl

Thank you @rafikdraoui, I missed that. Updated the README.


Add e2e tests for metadata

What does this PR do?
Add e2e tests for metadata
Motivation
Better test coverage
Additional Notes
Test passed locally:
$ ddev env test snmp py38 -k test_e2e_core_metadata

tests/test_e2e_core_metadata.py::test_e2e_core_metadata PASSED

Need https://github.com/DataDog/integrations-core/pull/10663 to be merged first for CI to run.
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

@FlorianVeaux @gmarabout Could you re-approve this PR ? I had to change the base branch and merge with master.


Is there a set glass data functionality?

I want to be able to alter the glass data parameters, eg. the transmission data. In ZPL there is a function GETGLASSDATA.  But in my case I need sth like setglassdata.
I've searched the source code, but didn't see anything. Perhaps, I've missed it though.
Thank you.

Since I am quite desperate for this, I am willing to implement it myself, if it can be done, given some direction. Otherwise, I'll have to throw out all my code and redo it in zos api

Hi @echatziz,
I also cannot find an implementation of that funcition, maybe @indranilsinharoy has an idea but I guess you will have to do the implementation yourself. I estimate that it it possible. I guess that you can make a copy of any function that gets data through the DDE client, and change the string command to get this data out. You can also pass parameters through thus string command, you can have a look at e.g. zGetTextFile() in zdde.py. Does that help you?

From what I've seen in the Zemax manual, although there is a ZPL command GETGLASSDATA, there isn't an ITEM with this functionality. Where ITEM is the function passed to the ZEMAX server. So, I guess it cannot be implemented, unless you can define your own item somehow.

Unfortunately, then I guess it is not possible this way. How many times do you need to change the glass data? Otherwise you might be able to make a catalogue by hand (or by using the zos-api) and then selecting it by zGetGlass() and zSetGlass()


Can't create Gpio on Raspberry Pi 4 8GB with 2020-08-20-raspios-buster-arm64

Working with this OS I can't create Gpio::new()?;, getting this error
Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })

~I've tracked the file to src/gpio/ioctl.rs:38~
~On source code there is~
const PATH_GPIOCHIP: &str = "/dev/gpiochip";

~but on my system there are these~
$ ls /dev/gpiochip*
/dev/gpiochip0  /dev/gpiochip1

EDIT: My bad, the Id is added later. Still the problem is on function find_gpiochip. While iterating it passes id 0 and id 1 and panics with id 2.
I'm using a raspberry pi 4 8GB
$ uname -a
Linux pocketvac 5.10.5-v8+ #1392 SMP PREEMPT Sat Jan 9 18:56:30 GMT 2021 aarch64 GNU/Linux

EDIT 2: Apparently the function is skipping gpiochip0 because the driver is pinctrl-bcm2711 instead of the expected pinctrl-bcm2835

Same issue here using 4B 4GB with 5.10.13

+1 to the issue

If this is blocking you, as it is for me, one could temporary use: rppal = { git = "https://github.com/ryanfuji/rppal.git", branch = "patch-1" } in your Cargo.toml. This points to #70.

The current master branch has been updated to recognize all currently available Raspberry Pi models. You should be able to use that in your projects until the next release. Let me know if you run into any additional problems.
It has been a while since I've had time to work on this project, and I need some time to sort out any remaining required changes, and update the documentation. 0.12.0 should be good to go soon though.


Add asyncSpawn action

I'm not sure how you'd like to handle using spawn. I went with a separate option so it's not a breaking change.
Once the api is settled on, I'll get a unit test added.
Fixes #43

I updated the spawn code to replace exec. I haven't had luck getting stdout returned as the resolve though (hence the failing test). I have very little experience with stdout/stderr and the impact the options for stdio have.  So if you could take a look at that and let me know how best to proceed that'd be awesome.

@bmcallis Great! Merged into v3.1.0 will do release when I solve the buffer thing.


Intel 8080 Emulator for the Atari 130XE

Hi,
This is the Atari 130XE 8080 emulator overlay loader. I did not include the whole source tree of https://github.com/ivop/atari8080 but just the assembled overlay and a few 8080 sample files. If you want Zork dropped because of licensing issues, I can create a new pull request. Do you know any other 8080 only software that could be included? Most interesting stuff I found is Z80 only.
Here's a small video demonstrating how it works:
https://youtu.be/oHrtUj0fEk0
I briefly looked into porting it to the BBC. I think it could be fairly straight forward with the 16kB banking window at $8000-$bfff. Incrementing the 8080 program counter will be slightly slower though, because the bpl trick cannot be used. The Atari's window is at $4000-$7fff, so then the high byte overflows, the N flag is set. On the BBC you'll need to load it into A, compare with $c0 and use bcc instead of bpl. That'll especially affect short instructions, like MOV, but ALU stuff not so much. Also, the banking register and its values have to be changed and setting/resetting the background color. The rest is platform agnostic. Oh, and the handling of overflowing the bank during a sector read and write needs to be handled differently, as there is no RAM after the bank like on the Atari. And the overlay loader needs to be changed slightly, e.g. check for BBC extended memory instead of the 130XE scheme.
I'm not sure I know enough about the BBC to do this myself, so I think you are better equipped to do this.
Regards,
Ivo

I looked a bit more on porting it to the BBC and noticed the following:

MOS cannot read/write directly to/from sideways RAM. That'll mean that the overlay loader cannot load BDOS and BIOS directly to the memory banks and that the BIOS translation for READ and WRITE need to be changed to copy the data (take care of crossing bank borders) to/from a buffer outside of the memory bank.
Mode 3 takes a lot of memory. Only the Master 512 has enough free TPA to run the emulator.

I also briefly looked into the Apple ][ and I don't think a port would be viable. It banks the full lower 48K, banking out the 8080 registers and the emulator, so memory access would be really slow by having a piece of code somewhere in the 4kB banks aboce $c000. There are Z80 cards for the Apple. Better use one of them.
I also got a request to port it to the Neo6502. I asked how banked memory was done or if it's even available, but got no response. I suppose the RP2040 firmware could be updated to support 64kB of banked memory, preferably at $4000-$7fff so the N flag trick keeps working.
Lastly, I thought of a way to improve emulation speed by eliminating the instruction size table and moving loading the operands and increasing the program counter to each instruction that needs it. This would make the main emulation loop much faster by eliminating at least three instructions for the single byte opcodes, and double or triple that for the two and three byte opcodes. For two bye opcodes, I can probably even remove the intermediate storage in byte2 and use (PCL),y directly, saving two more instructions. Three byte opcodes will have to keep the ZP storage because of dereferencing it as an (adjusted) pointer.

That's really impressive. The performance is better than I was expecting --- have you tried something like WordStar on it? I suspect it won't update fast enough to be useful, but maybe...
Re the BBC: yes, the high resolution modes are basically a non-starter on a stock B with CP/M-65. You really need a Master or, preferably, a Tube system. The latter uses a completely different CPU to run user programs, with its own 64kB of memory, and system calls are sent via a fast link via RPCs. However, the Tube doesn't have banking so it wouldn't work for you.
Regarding the banking addresses... the BIT instruction loads bit 6 of the target into V. You know bit 7 will always be set, so a BIT/BNE will detect a change from $bfff to $c000. It'll corrupt A, but ought to be cheaper than an LDA/CMP depending on circumstances.
Re the RP2040: the Morpheus firmware package (the one with NeoBasic in it) only has space for the 64kB of main RAM and 20kB of graphics RAM. Last time I looked there was about 16kB free. To get more you'd have to change the firmware to remove stuff like the graphics subsystem, but that's a step down the slippery slope that ends up with emulating the 8080 on the RP2040... you could do banking via external RAM, like SPI-attached PSRAM, but again that would need a firmware change.

Thanks! I must say I was surprised by its performance, too. I currently have a 'faster' branch in the atari8080 repo which is even 10-15% faster.
Re the BBC, yeah I guessed the Tube wasn't able to use the sideways RAM banks. So only the Master 512 would be a viable option with screen memory in shadow RAM, and the 8080's 64kB in 4 banks of sideways RAM. The BIT trick is indeed useful. AFAIR BIT does not clobber A, so this should work:
(in INCPC macro):
inc PCH
inc PCHa       ; adjusted high byte so (PCL) points inside extended memory bank
bit PCHa
bvc no_adjust
; recalculate PCHa from PCH and switch to next bank
no_adjust:
That's only one instruction longer than the Atari version. And even if A was clobbered, that doesn't matter for the code following.
Re Neo6502, I'll tell the person that asked that it's not possible :smile:
I found a website with WordStar 1.0, 2.26, and 3.0 and onwards. It seems 3.0 introduced the Z80 dependency, so perhaps 2.26 will run. Hopefully it does vt52 and not ADM 3a. I'll look into it.
I have to rerun the testsuite after my ~13% speedup to see if nothing broke.

WordStar 2.26 runs! But... the install.com utility to change the terminal type does not support VT52. You can enter your own values, but you need to know the offsets into the binary. It literally asks for a hex offset and values. Those are in the WS manual, but I cannot find a WordStar 2 manual :disappointed:
LOCATION TO BE CHANGED (0=END): 1234
    ADDRESS : 1234H   OLD VALUE: 41H   NEW VALUE:

I did a binary diff on two different WS.COM files. One for Zenith H89/H19 and one for Televideo 912 and it appears we can detect where the terminal codes are stored. Then match with the terminal manuals and see what they do and replace with VT52 equivalents. Hopefully that'll be enough?

I just ignored the Z80 label on https://winworldpc.com/product/wordstar/330 and installed WordStar 3.3. WINSTALL.COM supports entering the escape sequences for what it needs (cursor positioning and clear-to-eol) and with tty80drv and vt52drv loaded it runs! No Z80 code at all.
But it's too slow to do any real work with it. Loading a file works, cursor movement works, but it's not responsive enough. With a CPU accelerator like Rapidus with a 65c816 at 20MHz it runs flawlessly though! I don't have the real hardware though. Perhaps I'll get one for my birthday :wink:

I looked into a BBC port again, and I think I can make it work for every 128kB machine (B+128, Master128,Master512). At first I didn't get enough free TPA with the B+ and Master 128, but I found out I can force it to use shadow RAM by switching to mode 131 :smile: After that, it should work. Tested bank switching with $f4 and $fe30. Found out $f4 is important as the DFS ROM is needed, which I suppose is exactly the reason why I can't read directly into sideways RAM. So switching banks will be one instruction longer, but in the end I think it will run faster than the Atari. The Atari is effectively running at around 1.35MHz when RAM refresh and screen DMA is subtracted. The BBC will run at a full 2MHz, so that's 45-50% faster. Curious to see how that works out.

You might be able to get away with only updating 0xf4 before making system calls. It's used to remember which language is currently bank in, so that after making a system call it can return to the right place. I have faint memories that interrupts store the current bank using a different mechanism.

The BBC Master 128/512 port now works.
Video: https://youtu.be/5uONG8YH2L0
As expected, it runs quite a bit faster than the 130XE version. Still too slow for WordStar 3.3. Its terminal handling is terrible and I suspect it wasn't particularly fast when run on original hardware to begin with.
There's no B+128K support, as that turned out to be not so easy to do in one binary. It would either need a re-assembled binary specifically for the B+128K (bank 0,1,12,13 instead of 4,5,6,7) or live patching itself, which are quite a few location to be patched. For now it's Master 128/512 only, in mode 131 with video memory in shadow RAM.
Because of lack of disk space, I have it create a second bbcmicro.ssd image with just the screen apps (for vt52drv). Similar to the Atari 130XE I put the 8080 loader, overlay and its 8080 binaries in user area 2. Even though this was never discussed, I sort of treat user area 0 for all cross-platform stuff, user area 1 for platform specific files (e.g. setfnt for the Atari) and user area 2 for 8080 CP/M. This minimizes the risk of accidentally running a .COM file for the wrong CPU and avoids filename clashes (e.g. DUMP.COM).

As several months have passed, what are your thoughts on this? Is it something you'd like to include in CP/M-65 or do you prefer it as a separate project? If it's the latter, I'll start releasing disk images at the atari8080 repo. People have been asking me for a bootable release :smile:

So I'm in two minds about this. It makes sense to have it as part of the CP/M-65 distribution, but I don't really want to have checked in binaries. I do actually have a Z80/8080 assembler which could be easily integrated. OTOH I don't really want CP/M-65 to become a CP/M-80 distribution too...
Would it be feasible to have just the emulator stuff here, and then require a CP/M-80 disk obtained from elsewhere? Or does that just overcomplicate things?

Yeah, I was conflicted about it, too. Didn't know exactly what to put in third_party, and what not. 8080.s could be in included as we already have the Mad-Assembler as a requirement to build Altirra  Basic. But it needs the CP/M-80 binaries to build (custom bios, bdos and ccp). As I didn't want to include the full build system of atari8080, I opted to include the binary blob and discuss later. Which is where we are now :)
So if we were to include 8080.s in the CP/M-65 build system, it either needs the 8080 bios.sys, bdos.sys and ccp.sys as binaries or build them from source, too. Once the emulator runs, it needs applications or it's useless. Halt.com would be mandatory so you don't have to reset the machine to return back to  CP/M-65. As far as I know, multi-drive support in CP/M-65 is not yet working 100%, or has that been solved? If that was working, user area 2 on the boot disk (A:) could just include the overlay loader, the overlay and halt.com and the rest could be user supplied on drive B:.
Or I could keep maintaining my 8080ovl branch and periodically build disk images and release them through the atari8080 repo. I wouldn't mind doing that, and I can include a full CP/M-80 base system to make it more useful out of the box.

Could the 8080 code be loaded from the boot track of the CP/M-80 disk, mimicking the way that CP/M normally boots? That would neatly divide the 6502 code from the 8080 code (and the emulated CP/M-80 build could be easily added to my cpmish distribution). You'd put the CP/M-65 disk in A: and the CP/M-80 disk in B: and the overlay would do the mapping.
Multidrive support should work now, I think? Someone recently submitted a patch for it. I agree it's irritating on the BBC Micro. One of the things on my very long to-do list is to adapt the neo6502 code to do BDOS emulation on top of the MOS native hierarchical filesystem. Then the BIOS could be changed to do real disk access, so giving access to ADFS files as well as block access to the whole disk. One problem there is that ADFS only supports 10-character filenames, so there'd need to be a lot of subdirectory nesting...

That's an interesting idea. In theory, either the overlay loader or the emulator itself could read sectors or even files directly from drive B:. I guess the easiest would be for the overlay loader to load bios.sys, bdos.sys and ccp.sys from drive B: and mimic how it currently loads them directly to banked memory (with a detour via normal memory on the BBC because it can't load directly into sideways RAM). Prerequisite would be that drive B: has the same CP/M-65 disk layout as the host as it uses the host BIOS and BDOS to load the files. Theoretically it could load BIOS from the boot sector, but that would needlessly complicate things IMHO.
The 8080 disk could be built with your cpmish if you want to go fully open source, or with https://github.com/ivop/cpm22-from-source/ , to run the original DRI code (which BTW uses your ASM reimplementation with a wrapper to run directly on Linux, and it also compiles the original PL/M code with a ported ISIS compiler).
I'll look into multi drive support shortly. The Atari SD drive port could use an extra drive or two, and the Ohio Scientific Mini Floppy (just 640kB per disk) that I have developed the last few days would definitely benefit from it, too. I'll send pull requests once I have that all working correctly.
As for the 8080 emulator, I'll revisit that when I'm done with my current projects. Too many things on my todo-list ;)

Just a note: There is small bug in atari8080 which I pointed here: https://github.com/ivop/atari8080/issues/3


#159054017 Follow/Unfollow profiles feature

What does this PR do?
Allows users to follow other users and unfollow them. It also enables user to get a list of people they follow and people following them.
Description of Task to be completed?

Write tests for the feature
Modify profile model to include a follows field with a many-to-many relationship with the model
Develop methods to follow and unfollow profiles
Include urls for the views
Develop views for the follow/unfollow function and get followers and following profiles

Endpoints

METHOD
ENDPOINT
FUNCTIONALITY

POST
/api/profiles/:username/follow/
Follows a profile

DELETE
/api/profiles/:username/follow/
Unfollows a profile

GET
/api/profiles/:username/followers/
Gets list of followers

GET
/api/profiles/:username/following/
Gets list of following

How should this be manually tested?

Clone this repo

cd ah-jarvis

git pull origin ft-follow-unfollow-authors-159054017

create and activate a virtual env

pip install -r requirements.txt

python manage.py migrate

coverage run manage.py test

When all the tests pass, it means that the feature works.

Any background context you want to provide?
None
What are the relevant pivotal tracker stories?
#159054017 Users should be able to follow each other

You should change the endpoints to have a single slash instead of 2

@DennisWanjiru I've made the correction.

@asamisellah  , add a token to the following methods in authors.apps.authentication.tests.test_profile.TestUserProfile

test_get_profile
test_non_existent_user

It will solve your error as the profile needs an authenticated user.


Howler 🎵

@njam3 will fix test file class name.
I wanted to keep only click & rollover and load test sounds from external URLs so that there's no crap in the assets but there are CORS issues when running locally.
I can do another sprite - click + rollover and also keep them as separate sounds as well. I think it's fine to have both button-rollover + button-click out of the box as they are most basic sounds that you could have.
How does it sound?

@namniak Sounds good to me

@njam3 done


Fix queueing a texture and regular uniform update in RD backend

Fixes bug in the RD renderer where if a texture update has been queued, and then a generic uniform update is also requested, this update would be lost due to the texture update still being queued. This should be fixed by simply allowing you to always update the material dirty flags before checking if a material is queued rather than after. This bugfix is nessecary for this PR to work (#56145) since it can trigger this behaviour. It likely fixes other similar bugs too.

Thanks!


[FLINK-18143][python] Fix Python meter metric incorrect value problem

What is the purpose of the change
This pull request fixes Python meter metric incorrect value problem.
Brief change log

Report the diff value instead of the total ones for meter.

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

@hequn8128 Thanks for the PR. LGTM.

@dianfu Thanks. Merging...


DragSource attempting to setup HTML5Backend a second time in v9.4.0

Describe the bug
After updating to v9.4.0 we found that when mounting components using the DragSource decorator, it would sometimes attempt to setup the HTML5Backend again, thus triggering the error Uncaught Error: Cannot have two HTML5 backends at the same time (as this.window.__isReactDndBackendSetUp was not false).
The Dnd Context Provider is declared in the root of the page (and thus is above the components that are unmounted and mounted in the tree).
I suspect that either the teardown isn't triggering when it should, or the setup is triggering when it shouldn't, thus triggering this error.
Also, it should be noted that having multiple DragSources (such as items in a list), doesn't trigger this when they're rendered together. It's only when the list is replaced with another list that this bug is triggered.
Expected behaviour
We expect that unmounting and mounting two components that use react-dnd to correctly update the this.window.__isReactDndBackendSetUp global variable without throwing an error.
Desktop (please complete the following information):

OS: macOS (although this has been reported on other systems as well)
Browser: Chrome
Version: 77

Additional context
The stack trace from console:
Uncaught Error: Cannot have two HTML5 backends at the same time.
    at HTML5Backend.setup (HTML5Backend.js:388)
    at DragDropManagerImpl.handleRefCountChange (DragDropManagerImpl.js:48)
    at Object.dispatch (redux.js:227)
    at HandlerRegistryImpl.addSource (HandlerRegistryImpl.js:100)
    at registerSource (registration.js:13)
    at DragDropContainer.receiveType (decorateHandler.js:154)
    at DragDropContainer.receiveProps (decorateHandler.js:139)
    at DragDropContainer.componentDidMount (decorateHandler.js:115)
    at commitLifeCycles (react-dom.development.js:20049)
    at commitLayoutEffects (react-dom.development.js:22813)

https://github.com/react-dnd/react-dnd/issues/1571

I am running into this type of issue as well, in my case I have some set of components mounted that uses react-dnd, and then say I open a modal with another type of component that also uses react-dnd and is wrapped with its own DndProvider, I encounter this error. Without wrapping the component I encounter other errors such as invariant found, etc. where it can't set up the drag/drop context.

@ozzyogkush Maybe you can try using a single DndProvider mounted at the root of your app - there should only be one per document.

I found a temp. solution that works as such:
(in my app:)
<DndContext.Consumer>
  {({ dragDropManager }) => {
    return <MyComponent dragDropManager={dragDropManager} />;
  }}
</DndContext.Consumer>

In my component:
function DndConsumerWithFallback({ dragDropManager, children }) {
    return dragDropManager === undefined ?
        <DndProvider backend={HTML5Backend}>{children}</DndProvider> :
        <DndProvider manager={dragDropManager}>{children}</DndProvider>;
}

function MyComponent(props) {
    return (
        <DndConsumerWithFallback dragDropManager={dragDropManager}>
          {/* STUFF THAT USES DND INTERNALLY */}
        </DndConsumerWithFallback>
    );
}

This way, if the app doesn't provide the manager, it falls back on a default.

@darthtrevino is the expectation that once the dnd provider component is mounted, it should never unmount ?

@pjgates I don't know how bad this idea is but you can make it work by setting yourself the property to false on mount.
useLayoutEffect(() => {
   window.__isReactDndBackendSetUp = false;
}, [])

I don't have much knowledge about working of react dnd and I would like to get feedback on this approach wether this will break code or not.

The DndProvider break down the HTML5Backend when it is unmounted, and the HTML5Backend will clear that variable in the window.
That being said, all of the testing of this library has been done with a stable root DndProvider in the sample apps

[HTML5BackendImpl.ts#L108-L108](https://github.com/react-dnd/react-dnd/blob/8e6f62e612c0b0a376ac4a86cd371588dfcd2a70/packages/backend-html5/src/HTML5BackendImpl.ts#L101-L112)
maybe this rootElement can be replace by ReactRef?
I cant use custom rootElement when my component is not mounted.
@darthtrevino


NullPointerException

I am using com.plattysoft.leonids:LeonidsLib:1.3.1 version of this library and getting the following crashes from google play store :
java.lang.NullPointerException
!   1   at com.plattysoft.leonids.ParticleSystem.onUpdate(ParticleSystem.java:556)
2   at com.plattysoft.leonids.ParticleSystem.access$100(ParticleSystem.java:35)
3   at com.plattysoft.leonids.ParticleSystem$1.run(ParticleSystem.java:363)
4   at java.util.Timer$TimerImpl.run(Timer.java:284)
So can you please check it and resolve it?

There is a lot of information missing here.
Is this a crash on the Demo app on Play Store? If so, on which example? And what are the steps to replicate. Is it 100% replicable?
I did re-check the examples right now and they seem to work fine.
If this is something you are getting on you own app, please post the code, you may be doing something wrong on the initialization of the Particle System.

Is this a crash on the Demo app on Play Store?
Ans : It is on play store.
It is not re-producible from my end but occasionally I am getting this error on google play store console and also getting this crash log on https://www.apteligent.com/

That was not my question. The question was if the crash was happening on the Demos, not if your app was on the Play Store.
Without steps to replicate, there's nothing I can do.

I have the same issue too. Not sure what's going on and here is the stacktrace from Crashlytics
Fatal Exception: java.lang.NullPointerException
       at com.plattysoft.leonids.ParticleSystem.onUpdate(ParticleSystem.java:556)
       at com.plattysoft.leonids.ParticleSystem.access$100(ParticleSystem.java:35)
       at com.plattysoft.leonids.ParticleSystem$2.onAnimationUpdate(ParticleSystem.java:445)
       at android.animation.ValueAnimator.animateValue(ValueAnimator.java:1251)
       at android.animation.ValueAnimator.animationFrame(ValueAnimator.java:1175)
       at android.animation.ValueAnimator.doAnimationFrame(ValueAnimator.java:1216)
       at android.animation.ValueAnimator$AnimationHandler.doAnimationFrame(ValueAnimator.java:637)
       at android.animation.ValueAnimator$AnimationHandler.run(ValueAnimator.java:660)
       at android.view.Choreographer$CallbackRecord.run(Choreographer.java:761)
       at android.view.Choreographer.doCallbacks(Choreographer.java:574)
       at android.view.Choreographer.doFrame(Choreographer.java:543)
       at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:747)
       at android.os.Handler.handleCallback(Handler.java:733)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:136)
       at android.app.ActivityThread.main(ActivityThread.java:5372)
       at java.lang.reflect.Method.invokeNative(Method.java)
       at java.lang.reflect.Method.invoke(Method.java:515)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:970)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:786)
       at dalvik.system.NativeStart.main(NativeStart.java)

@Smeet123 I figure out the for my case. I've been using a View.post() to force the animation to run on UI thread. I also have a destroy() call to .cancel() in ParticleSystem. It is very likely that the Runnable in my View.post() function is executed after .cancel(), where mDrawingView will become null. See if that's your case too.

@raymondctc Thank you so much for the solution, I will check it once I get time.

May not be the cause of the reported NPE, but I can see a data race in ParticleTimerTask.run():
    if(mPs.get() != null) {
        ParticleSystem ps = mPs.get();

The two lines should be inverted:
    ParticleSystem ps = mPs.get();
    if(ps != null) {

The same


chore(aurora, removal): remove aurora from the build system

First step! - This should remove Aurora from the build system entirely. Does not yet modify any other files since I'd like to handle this step-by-step

It built fine locally, pretty sure it can be merged just fine now


GitHub action run on all branches

This is needed for release 1.4.5, will merge so we can get other PRs in before the weekend. Thanks @pranavgaikwad


API spec and API functionality disparity

I'm going through #13 again, and I'm still a bit confused about what to do if the API spec does not match the API functionality. Often, it may be me now knowing how to work Postman. Sometimes, however, the disparity is caused by a poorly formed API, I think.
For instance, ipfs bitswap unwant should return at 204, not a 200. It doesn't return anything. Should I leave this as a 200, or change it to a 204 and open an issue in go-ipfs?

From Sprint of January 5th call -> https://github.com/ipfs/pm/issues/77#issuecomment-169143978

Let's have the current spec, which describes what is implemented and have the PR's to the current spec with the changes that make sense for each endpoint and open issues in go-ipfs to fix those endpoints

Thanks for posting that here. This can be closed. Current workflow:

Add in all current functionality, broken or not, in apiary format. This involves drawing from the draft on feature/add and PRing each method into master.
Get feedback on issues in the PRs into master.
Merge PRs into master, then PR separate issues where functionality does not match spec (currently undefined), with links to issues on go-ipfs which would enable PRs to be merged.


Add an SAPUI5 App to Your Launchpad Site

Tutorial: https://developers.sap.com/tutorials/cp-portal-cloud-foundry-new-sapui5.html
Ubuntu 18.04.5 LTS
left chorme   / right  firefox 84
In firefox its no possible create/edit  or  delete   content .(content-manager)
https://user-images.githubusercontent.com/61939539/102702020-80203f00-423c-11eb-8ddc-6aca4cccf427.mp4

Hi - I don't know what the problem is.
Can you open a ticket for this issue. Perhaps someone in the community can help.
Thanks
Lindsay

These GitHub issues are to resolve issues (i.e., problems/errors) with the tutorials. Your question is more general and likely would get more visibility -- and a better answer -- in the SAP Community's Questions & Answers.
https://www.sap.com/community.html
https://answers.sap.com/index.html
@LindsayBert  Use this test and these direct links if you need to refer someone to the community.


Add support of skipping tests that are not present in testrun

I think that it would be greate, if you will be able to skip tests, that are not present in testrun.
Now they are made but not published - for my number of testcases it may lead to a lot of time loose.
Example param:
--tr-skip-missing

@allankp Now waiting here for you upprove ASAP. :smile:
Need it in project. Now just using it hardcoded.

Hi! Is it possible to add the same functionality for TestPlan?


Cleanbot tweaks

Now they actually mop floors.
I had an idea on how to make NPC queries less shit so the pick operators are gonna be similar for a bit until I flesh that out.
I buffed it to 10 absorbent per interaction with a delay of 2s (5 / 1.5s before) after recording the video.
https://user-images.githubusercontent.com/31366439/234558844-869ab61c-869d-425a-9e84-d2835351116d.mp4
The target selection is also bad due to the aforementioned query implementation.
:cl:

tweak: Cleanbots now mop floors rather than drain near them.

cleanbot my beloved


Correct the tense of "known". Typo in a comment. [2nd submission]

Hi Bram and all! You closed my first pull-request for this branch with a comment that you'll merge it but didn't in over 20 days. This is the second take and please merge it properly this time. Meanwhile, the corresponding pull-req for neovim got merged and is part of its main repository.

Codecov Report

Merging #1684 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1684      +/-   ##
==========================================
+ Coverage   74.85%   74.88%   +0.03%
==========================================
  Files          76       76
  Lines      124942   124942
==========================================
+ Hits        93521    93565      +44
+ Misses      31421    31377      -44

Impacted Files
Coverage Δ

src/buffer.c
75.15% <ø> (ø)
:arrow_up:

src/message.c
68.07% <0%> (-0.21%)
:arrow_down:

src/os_unix.c
58.29% <0%> (-0.05%)
:arrow_down:

src/channel.c
83.62% <0%> (-0.05%)
:arrow_down:

src/getchar.c
73.84% <0%> (+0.04%)
:arrow_up:

src/screen.c
74.27% <0%> (+0.15%)
:arrow_up:

src/term.c
53.37% <0%> (+0.17%)
:arrow_up:

src/gui_gtk_x11.c
47.39% <0%> (+0.2%)
:arrow_up:

src/window.c
81.12% <0%> (+0.53%)
:arrow_up:

src/if_xcmdsrv.c
83.88% <0%> (+0.55%)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4475b62...ca60cc9. Read the comment docs.

The change was merged, just didn't have another change in this file yet.


Why command for Paket v3

All the changes for Why command rabased on master branch.

docs for the command
handle circular dependencies
reword "transient" ==> "transitive"

Amazing. Thanks so much

you're welcome ;)


Getting operation not permitted when tests from workflow attempt to modify or remove a file

Description
We have a test that modifies a file and test that removes a file. It is related to our CLI and it's context file.
Our test fails (randomly, but more often fails that is successful) with EPERM: operation not permitted, open 'D:\a\cli\cli\test.asyncapi'
For example this PR fails here even though it is not really adding the failing tests, they are already there. Interesting is that this doesn't happen with other PRs.
We tried to modify CLI to have custom context file location, so it is not on C drive but where the workflow runs, but this didn't solve the problem.
Any ideas?
Virtual environments affected

[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[ ] macOS 10.15
[ ] macOS 11
[ ] Windows Server 2016
[X] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
  Environment: windows-2019
  Version: 20220116.1
  Included Software: https://github.com/actions/virtual-environments/blob/win19/20220116.1/images/win/Windows2019-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/win19%2F20220116.1

Build: https://github.com/asyncapi/cli/runs/4864670498?check_suite_focus=true
Is it regression?
yes
Expected behavior
it should pass test
Actual behavior
provided ☝🏼 in description
Repro steps
Random so not reproducable. We have no idea why it is happening exactly in this PR randomly, issue with tests that were not modified

Hello @derberg. We'll take care about this issue investigation and reply with findings.

Hi @derberg, as far as I can see, test.asyncapi file is created by fs.writeFileSync() and options.flag is w by default. According to File system flags documentation:
On Windows, opening an existing hidden file using the 'w' flag (either through fs.open() or fs.writeFile() or fsPromises.open()) will fail with EPERM. Existing hidden files can be opened for writing with the 'r+' flag.
I'm going to close this issue since it's not related to image generation. Please let us know if you have any objections.

@nikolai-frolov we will test it out, but it will be super strange if that works. Tests fail randomly, and only on one PR 🤔

@nikolai-frolov please have a look at below discussion from linked PR

@derberg could you please test w+ flag instead and/or async version of function? As I can see from error stack, file was created but can't be opened, so as an additional debugging I would suggest check if file actually exists and it's access attributes.

@derberg as far as I can see, the issue root was found in the project code, so I'm going to close this issue. Feel free to reach us in case of any issues.

@nikolai-frolov yes, sorry for bothering you 😞 🙏🏼 and thanks a lot for taking time to investigate it with us 🙇🏼


Add support for component box

http://camel.apache.org/box.html

Maybe best to wait for Camel 2.19 before integrating this one. The box component in 2.18.x has been deprecated and rewritten from scratch in 2.19.

Found 22 matching posts for subject:box in Camel - Users within 700 days Showing posts 1 to 12.


Mysterious response for input on POST /beacon/validators

This request:
curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"epoch": "0", "pubkeys": ["0x1b66ac1fb663c9bc59509846d6ec05345bd908eda73e670af888da41af178bb2305b26a285fa2737f175668d0dff91cc"]}' localhost:5051/beacon/validators

Produces this response:
{"status":400,"message":"The serialized input does not have the C flag set. (through reference chain: tech.pegasys.teku.api.schema.ValidatorsRequest[\"pubkeys\"]->java.util.ArrayList[0])"}

Which is rather mysterious.

it looks like this is not a valid key because G1Point is raising an error about not having the 'c flag' set, but i think we need to wrap this message and report back that the public key supplied is not valid.

You're right, that pubkey was no good. Using a different one works.

error message should include key that was bad

error message includes index of key that was bad


Fix DockerInstallerV0 for ARM agent

Task name: DockerInstallerV0
Description: Added case for armhf based docker installer
Documentation changes required: N
Added unit tests: N
Attached related issue: https://github.com/microsoft/azure-pipelines-tasks/issues/13198
Checklist:

[x] Task version was bumped - please check instruction how to do it
[x] Checked that applied changes work as expected

@ds-ms I would very much like to add a test but this is beyond my current ramp-up on the azure-pipelines-tasks framework.
I've looked into the tests file and there's a big TODO in there (no tests at all). If anyone else would like to contribute with that, I'd be happy to follow along. I've looked into the other installers (helm and kubectl) for sample tests but couldn't find any.
It would have been great if a test was added when arm64 was introduced so I had something to follow as an example of the framework.
My fix is is for adding a missing option for an architecture which is marked as "supported".

Oh! My bad. I just saw the L0.ts file to be present and expected the test framework to be setup correctly, already.
Thanks for the contribution. It LGTM; I'll take it forward. 😄

The rollout for this would start next Monday.. should reach all the rings within the next 3 weeks


Remove 3bot from infra solutions

The only way to get a 3bot is from deploy3bot.grid.tf

Commit_ID 438c93ec11e014904f800d21d42019f807e000bc, Branch development


[FEATURE REQUEST] Novel Updates x Blogspot

Many novels are on blogspot, but there is usually no way to download them. I noticed that novelupdate collects the chapter links and keeps them up to date so I'm wondering if it's possible to cobble together a feature that allows you to collect all the links and then download the novel. I don't know if it's possible and that's why I ask.

Manual grabbing was designed for this.

I see... I will check that. Thanks.


System.ArgumentOutOfRangeException when generating objects from a JSON schema

Environment

Nuget Package: JsonSchema.Net.DataGeneration
Nuget Version: 1.0.1
OS: macOS
.Net Target: .NET 6

Describe the bug
When generating data from a schema that defines an object with required properties, the following exception is thrown from the Bogus library...
System.ArgumentOutOfRangeException : Specified argument was out of the range of valid values. (Parameter 'count')
   at Bogus.Randomizer.ArrayElements[T](T[] array, Nullable`1 count)
   at Json.Schema.DataGeneration.Generators.ObjectGenerator.Generate(RequirementsContext context)
   at Json.Schema.DataGeneration.JsonSchemaExtensions.GenerateData(RequirementsContext requirements)
   at Json.Schema.DataGeneration.JsonSchemaExtensions.GenerateData(JsonSchema schema)
   at JsonSchemaReproTests.Tests.Test1() in /Users/aaronlahey/Desktop/JsonSchemaRepro/JsonSchemaReproTests/UnitTest1.cs:line 24

In the Bogus library, an exception is thrown when generating a random set of array elements if the requested count is greater than the size of the array...
https://github.com/bchavez/Bogus/blob/v34.0.2/Source/Bogus/Randomizer.cs#L512-L513
In this lib the code is passing Math.Min(definedPropertyNames.Count, remainingPropertyCount) for the count but it is passing the propertyNames array which can lead to the requested count exceeding the length of the array.
https://github.com/gregsdennis/json-everything/blob/schema-datageneration-v1.0.1/JsonSchema.DataGeneration/Generators/ObjectGenerator.cs#L93
~It seems like the error case is triggered when generating objects that have N properties where N > 2 and R required properties where R = N - 1.~ See the following comment for a test case.
To Reproduce
The smallest test case appears to be something like this...
var schema = JsonSchema.FromText(@"
  {
    ""type"": ""object"",
    ""properties"": {
      ""1"": { ""type"": ""number"" },
      ""2"": { ""type"": ""number"" },
      ""3"": { ""type"": ""number"" }
    },
    ""required"": [""1"", ""2""]
  }
");
for (var i = 0; i < 1000; i++)
    schema.GenerateData();

Expected behavior
No exception.
Additional context
See below.

For more context, here is another test I ran...
for (var numProperties = 0; numProperties < 25; numProperties++)
{
    for (var numRequiredProperties = 0; numRequiredProperties <= numProperties; numRequiredProperties++)
    {
        var properties = new JObject();
        for (var i = 0; i < numProperties; i++)
        {
            properties[i.ToString()] = new JObject
            {
                ["type"] = "number"
            };
        }

        var required = new JArray();
        for (var i = 0; i < numRequiredProperties; i++)
        {
            required.Add(i.ToString());
        }

        var schemaJObject = new JObject
        {
            ["type"] = "object",
            ["properties"] = properties,
            ["required"] = required
        };

        var schemaJson = JsonConvert.SerializeObject(schemaJObject);
        var schema = JsonSchema.FromText(schemaJson);

        try
        {
            for (var i = 0; i < 1000; i++)
            {
                schema.GenerateData();
            }
        }
        catch (ArgumentOutOfRangeException e)
        {
            Console.Error.WriteLine($"Fails when there are {numProperties} properties and {numRequiredProperties} of them are required.");
        }
    }
}

Output:
Fails when there are 3 properties and 2 of them are required.
Fails when there are 4 properties and 3 of them are required.
Fails when there are 5 properties and 3 of them are required.
Fails when there are 5 properties and 4 of them are required.
Fails when there are 6 properties and 4 of them are required.
Fails when there are 6 properties and 5 of them are required.
Fails when there are 7 properties and 4 of them are required.
Fails when there are 7 properties and 5 of them are required.
Fails when there are 7 properties and 6 of them are required.
Fails when there are 8 properties and 5 of them are required.
Fails when there are 8 properties and 6 of them are required.
Fails when there are 8 properties and 7 of them are required.
Fails when there are 9 properties and 5 of them are required.
Fails when there are 9 properties and 6 of them are required.
Fails when there are 9 properties and 7 of them are required.
Fails when there are 9 properties and 8 of them are required.

Thanks for the report.  I'll have a look.

Since JsonSchema.Net is ina a prerelease for the next major version, the compiler is warning that a non-prerelease package shouldn't depend on a prerelease package.  I'm going to try to apply this to a previous commit (before the JsonSchema.Net update) and see if I can publish from there.  I've never done this, and I'm not really set up for it, so let's see how this goes.

Nice!  Looks like the publish worked!
V1.0.2 should resolve this issue.


browserify support: Can you publish this on npm?

Hi @jsdevel! Sure I can take care of that. I'll let you know. Cheers.

Thanks!

Hi @jsdevel!
I needed also to add an index.js file before publishing it. Thanks for commenting and using the filter.
Cheers.

Thanks @Puigcerber !


[Feature-WIP](inverted index) support array type for inverted index reader

Proposed changes
Issue Number: Step4 of DSIP-023: Add inverted index for full text search
support array type for inverted index reader
dependency pr: https://github.com/apache/doris/pull/14211 https://github.com/apache/doris/pull/15823
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.53 seconds
load time: 520 seconds
storage size:<PHONE_NUMBER>4 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230202075550_clickbench_pr_89308.html


Update _element.py

please don't change the change.rst

ok

if you accept my pull request for _element.py i wouldn't do any chage at all

I think get_elements  is enough, but thanks


Add mob retaliation

About the PR

Mobs now fight back when you hit them. They'll give chase and try to beat you up a bit.
You can adjust how long they "stick" to you on the NPCRetaliation component.
Why / Balance

Gives mobs a bit more character.
Technical details

adjusts some of the code in FactionExceptionSystem to support hostiles. Mostly underlying.
Media

https://github.com/space-wizards/space-station-14/assets/98561806/81250a96-f02f-4ede-aac1-6eff7eee6822

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

:cl:

add: Certain mobs will now fight back when provoked. Try not to get robusted.

thank yoyu emo


Is it possible to use read only C++ FASTER in iOS and Android?

Is it possible to use read only C++ FASTER in iOS and Android?
The usage should be read only database - the database
is generated once and used in read only mode in the
smartphone.
The scenario is this:

generate the FASTER database in OS X
add C++ library and generated database in iOS or Android

You can possibly generate the database on the server in OSX, then take a checkpoint. In the Android/iOS client, you can restore that checkpoint and issue your read workload. It should work, but we've never tried it before.


DataReceived is not triggered

Hi,
i have a problem triggering the DataReceived Event. I installed virtual serial port tools (from HHD) and created a local bridge between COM2 and COM3. I created two small TestApps. One connects to COM2 and sends a string. The other connects to COM3 and receives data.
SerialTest:
`using System;
using RJCP.IO.Ports;
namespace SerialTest
{
class Program
{
static void Main(string[] args)
{
SerialPortStream serialPortStream = new SerialPortStream("COM2", 115200, 8, Parity.None, StopBits.One);
serialPortStream.Open();
if (serialPortStream.IsOpen)
{
Console.WriteLine("Port is open");
string sendString = "blablabla";
serialPortStream.Write(sendString + 0x1A);
Console.WriteLine("Data sent");
}
else
{
Console.WriteLine("Port is not open");
}
}
}
}`
SerialTest2:
`using System;
using RJCP.IO.Ports;
namespace SerialTest2
{
class Program
{
static void Main(string[] args)
{
string readBuf = null;
SerialPortStream serialPortStream = new SerialPortStream("COM3", 115200, 8, Parity.None, StopBits.One);
serialPortStream.DataReceived += delegate(object sender, SerialDataReceivedEventArgs eventArgs)
{
Console.WriteLine("sender = {0}, args = {1}", sender.ToString(), eventArgs.ToString());
};
serialPortStream.Open();
if (serialPortStream.IsOpen)
{
Console.WriteLine("Port is open");
            while(string.IsNullOrEmpty(readBuf))
            {
                // Do nothing
            }

        }
        else
        {
            Console.WriteLine("Port is not open");
        }
    }
}

}`
I start SerialTest2 which has a delegate on the DataReceived event. Then I start SerialTest1 which sends the string (including the EOF). The delegate in SerialTest2 is not invoked.
Do I use the SerialPortStream the wrong way?
Best
Mic

Please test again using 2.4.0 and report your results. Thank you!

Closing as there is no feedback.