cannot update highlight.js to a non-vulnerable version

Version
4.5.9
Environment info
Environment Info:

  System:
    OS: macOS Mojave 10.14.6
    CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
  Binaries:
    Node: 14.15.1 - /usr/local/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.8 - /usr/local/bin/npm
  Browsers:
    Chrome: Not Found
    Edge: Not Found
    Firefox: Not Found
    Safari: 14.0
  npmGlobalPackages:
    @vue/cli: 4.5.9

Steps to reproduce
use dependabot in your GitHub repository whilst using highlight.js and vue-cli (versioned 4.5.9).
What is expected?
I was expecting to use the highest version of highlight.js and thus fix a moderate security vulnerability.
What is actually happening?
It soon warns me of a moderate security vulnerability which has the impact of Potential ReDOS vulnerabilities (exponential and polynomial RegEx backtracking). I, as a perfectionist, wants to change this situation.
However, it tells me:
Dependabot cannot update highlight.js to a non-vulnerable version
The latest possible version that can be installed is 9.18.5 because of the following conflicting dependency:

<EMAIL_ADDRESS>requires<EMAIL_ADDRESS>via<EMAIL_ADDRESS>The earliest fixed version is 10.4.1.

It is a private repo, so I am sorry you cannot see the complete log.

I realised just now when choosing my version – is there a 5.0.0 alpha?
That might be the very key.

Maybe you should bump the version of cli-highlight to 2.1.9 which depends on highlight.js@10 by yourself. It seems dependabot can't handle this case.

Maybe you should bump the version of cli-highlight to 2.1.9 which depends on highlight.js@10 by yourself. It seems dependabot can't handle this case.

this should solve the issue as it does for me. Thanks!

this should solve the issue as it does for me. Thanks!


get_params and set_params fixed / cross validation test case added

I can't really test the cross validation itself, though cloning (sklearn.base.clone) works. Python gets a SIGKILL from somewhere. Maybe something on my side is wrong, should work though. Other than in the other pull request the deep copying of classifier (i.e.) works. (hasattr(getattr(self, attr), 'get_params') is the correct one)
See #29 for reference.

FYI: this is the output of the params:
>>> c2.get_params(deep=True)
{'classifier__bootstrap': True, 'classifier__max_features': 'auto', 'classifier__max_depth': None, 'classifier__min_samples_split': 2, 'classifier__min_weight_fraction_leaf': 0.0, 'classifier__oob_score': False, 'classifier__class_weight': None, 'classifier__n_estimators': 66, 'classifier__max_leaf_nodes': None, 'classifier__random_state': None, 'classifier__min_samples_leaf': 1, 'classifier__warm_start': False, 'require_dense': [True, True], 'classifier__n_jobs': 1, 'classifier': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=None, max_features='auto', max_leaf_nodes=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, n_estimators=66, n_jobs=1,
            oob_score=False, random_state=None, verbose=0,
            warm_start=False), 'classifier__verbose': 0, 'classifier__criterion': 'gini'}

for the classifier:
c2 = BinaryRelevance(RandomForestClassifier(n_estimators=66))

Needs further testing.

c2.set_params(**c1.get_params())

c2.set_params(**c1.get_params(deep=True))

c2 = clone(c1)

Do all work now.
Context:
from skmultilearn.meta import br
from sklearn.ensemble import RandomForestClassifier
from sklearn.base import clone

c1 = br.BinaryRelevance(RandomForestClassifier(n_estimators=66))

@niedakh From my point of view this can be merged now.

Ive got two busy days (sending a conference paper to Berlin), I'll take
take a look at this tommorow evening, sorry for the delay!

No problem. Good luck with the paper then ;)

thank you Christian! merged it.

I will convert your notebook to a unittest tommorow and add it to all the classifiers we have.

Awesome, thank you.


Remove optional fields throughout protos

The protoc-gen-gocosmos tool, used to generate golang code from protobuf definitions for many Cosmos projects, does not support the use of optional fields in proto3 syntax. We encountered this on https://github.com/strangelove-ventures/interchaintest/issues/644#issuecomment-1677750362. To unblock, let's drop use of optional fields, updating code where necessary.

Still very much in-progress. Modifying the governance proposal code not to use options got me all tangled up. Had a helpful conversation with @plaidfinch, recommending deeper proto changes that will accommodate, but I expect I'll need more support before this is done. Still a couple more things to band on before I ask for help again, though.

Paired with @aubrika today, who pointed out some comparable patterns for de-optionalizing. Used that approach to handle conversions of memo ciphertexts, bonding options, and how commit strings are handled in signaling proposals. Still outstanding however is sane handling of the "withdrawn" proposal state in the governance module. Planning to ask @plaidfinch for another pairing session to get that over the finish line.


Configure Extension Method

I downloaded IdentityManager.MembershipReboot & IdentityManager. I'm using the latest version of both.
However, I can't find the configure extension method. It says that there is no extension method "Configure" for IdentityManagerServiceFactory.
app.Map("/admin", adminApp =>
            {

                var factory = new IdentityManagerServiceFactory();
                factory.Configure(connectionString);

                adminApp.UseIdentityManager(new IdentityManagerOptions()
                {
                    Factory = factory,
                    SecurityConfiguration = new HostSecurityConfiguration()
                    {
                        HostAuthenticationType = "Cookies",
                        NameClaimType = Constants.ClaimTypes.Name,
                        RoleClaimType = Constants.ClaimTypes.Role
                   }
                });
            });

It's in the sample host project.

Got it thanks it works for me.


error:  class_parents(): Class 1 does not exist

How I can solve this?

he pleas open a new issue. When you have new errors.


Fix useDotnetFromEnv's DOTNET_ROOT detection

Description of changes
Closes https://github.com/NixOS/nixpkgs/issues/249075
This PR tries to fix the detection of locally installed dotnet versions, when using a package built with useDotnetFromEnv = true;
(Original PR introducing this feature: https://github.com/NixOS/nixpkgs/pull/235041)
The issue is that the original PR's detection was trying to find the correct DOTNET_ROOT from the currently installed binary, by running which dotnet and going up some levels in the parent directories.
I found that doing that is not correct because which dotnet will result in the symlink's path (/run/current-system/sw/bin/dotnet or /etc/profiles/per-user/<user>/bin/dotnet)
Using realpath, however, we can reach the /nix/store location of the program.
In dotnetCorePackages.sdk_6_0 (aka. dotnet-sdk) the actual binary is located at $out/dotnet. There is also $out/bin/dotnet which is just a symlink to the actual.
So running realpath $(which dotnet) will result in $out/dotnet which is great.
(realpath will follow a symlink chain until it reaches a normal file).
However when using dotnetCorePackages.combinePackages to have multiple sdks installed, realpath $(which dotnet) will result in $out/bin/dotnet, because $out/bin/dotnet is not actually a symlink, but the result of running wrapProgram. The wrapper adds dotnet-sdk.icu to the LD_LIBRARY_PATH.
(PR where this was added: https://github.com/NixOS/nixpkgs/pull/187118)
My fix was to remove this wrapper, so realpath can find the actual binary.

I am not yet sure why the wrapping with icu was needed if the non-combined binary didn't have it in the LD_LIBRARY_PATH.
@GGG-KILLER, as the original author of the commit with the wrapper, can you tell us why the wrapping was needed?
I will look for any packages where this removal would cause an issue.

If this PR is merged, we will need to make sure that for every installable dotnet there is an uninterrupted symlink-chain to $out/dotnet.
AFAIK, sdk_x_0 runtime_x_0, aspnetcore_x_0, and their combinations with combinePackages now all conform to this pattern.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review run on x86_64-linux 1

46 packages built:

ArchiSteamFarm
BeatSaberModManager
alttpr-opentracker
audiobookshelf
avalonia-ilspy
boogie
btcpayserver
btcpayserver-altcoins
certdump
csharp-ls
dafny
denaro
depotdownloader
discordchatexporter-cli
eventstore
fable
formula
fsautocomplete
galaxy-buds-client
git-credential-manager
ilspycmd
inklecate
jackett
jellyfin
kavita
marksman
mqttmultimeter
msbuild
nbxplorer
netcoredbg
networkminer
omnisharp-roslyn
openra
opentabletdriver
osu-lazer
pbm
pinta
ps3-disc-dumper
roslyn
ryujinx
scarab
slskd
space-station-14-launcher
tone
wasabibackend
xivlauncher

@TomaSajt I believe this was added due to some old .NET versions (which are now EOL) not properly patching icu and then this fix might've leaked onto combinePackages, however I don't remember why this was needed anymore so if the removal doesn't cause any issues I think it's fair to proceed with it.

Oh wait, now I remember why I did it, it was because the original .NET packages already use makeWrapper however it moves the original binary outside of bin and we copy from there, so I am re-applying the patch done by buildDotnet in postFixup: https://github.com/GGG-KILLER/nixpkgs/blob/01dd4610f8aa23df5e80e7daf579797013e5d1c4/pkgs/development/compilers/dotnet/build-dotnet.nix#L86-L87

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1065


Stream to Tapo camera

Hi,
I've set up Stream Assist on my Tapo camera. It is supported by go2rtc and WebRTC for streaming the reply and the STT start media. Everything is working during 1-2 min. It reacts to the wake word and performs tasks using assist. After a while a click can be heard from the camera. After that, streaming to the camera (media player entity) stops working. It still reacts to the wake word, but the STT start becomes somewhat undetermined. By chance, it can sometimes understand the voice command.
The situation is corrected if I restart the go2rtc, or reload the Stream Assist configuration.
Please give some advice, why the two-way-audo streaming to the camera stops, and how to correct.

"no route to host" it's an operating system error. You have something wrong with your network settings.
Given the very rare IP address - you have a very complex LAN configuration.

Hi AlexxIT,
Finally I gave up with the Stream Assist... I wasn't able to stabilize it. The streaming to the camera through WebRTC media player always stopped working, so there was no feedback about detecting the wake word. In the Stream Assist I tried different stream configurations: direct RTSP link to the camera, link to the go2rtc stream and also camera entity, but without success. Even more, the Tapo integration also became somewhat buggy if go2rtc stream, WebRTC media player and Stream Assist were configured.
Important to know, that I use the camera with HA through the Tapo integration and my Synologi Surveillance station also consumes RTSP stream of it.
Regards,
Bithajcsar


What Optional Methods are we allowed to use?

Does anyone know which Optional methods we are and are not allowed to use? Also if anyone knows the reasoning behind why we're not allowed to use them, that would be great!
I remember having to avoid things like get isEmpty isPresent, or, orElse. I never really understood why either. I'm guessing it's to abide by Tell-Don't-Ask, but I don't really see how that applies to or or orElse.

or and orElse can be used just that in the PA, the prof set extra restriction but if not mentioned, or and orElse can be used

Does anyone know which Optional methods we are and are not allowed to use? Also if anyone knows the reasoning behind why we're not allowed to use them, that would be great!
I remember having to avoid things like get isEmpty isPresent, or, orElse. I never really understood why either. I'm guessing it's to abide by Tell-Don't-Ask, but I don't really see how that applies to or or orElse.

As in past papers, you will be told explicitly in the question. However, as in Lecture 5, you should (NOT must) not use .get(), .isPresent(), or .isEmpty() to break the encapsulation.

just code declaratively

Avoid using .get(), .isPresent(), or .isEmpty() to break the encapsulation in the finals

finals is over you can close this issue


Exposed extism module and added methods to get exported functions

This PR does a couple of things:

It exposes the extism module struct.
It adds methods to get any exported functions from the module.
It adds a struct to wrap the wazero function definition, allowing Extism to control the API, while also exposing some of the behaviors provided by wazero.
It removes the module.wasm bytes from the module struct. It was unused as far as I could tell and I don't see any reason (while it's unused at least) that we wouldn't want the GC to be able to garbage collect that slice, especially when we're dealing with large modules.

thank you! will review today

🙌 Thanks for the consideration and the reviews!

Thank you for taking the time @clarkmcc :)


feat(tidb): tidb split release 8.0 jobs

split  tidb release 8.0 jobs.

/hold
wait new branch release-8.0 to be created.

/unhold


fix: remove custom iterator implementation

and introduce a BlockIterator.
Signed-off-by: Harald Hoyer<EMAIL_ADDRESS>

Oh, we should be able to remove this now as well https://github.com/enarx/sallyport/blob/6f0925ae3af419aa4cecfb17fdb8bb3a12a3f591/src/host/mod.rs#L116

Oh, we should be able to remove this now as well
https://github.com/enarx/sallyport/blob/6f0925ae3af419aa4cecfb17fdb8bb3a12a3f591/src/host/mod.rs#L116

@rvolosatovs  done


nlink of trash dir should be 2.

What happened:
Script:
     META_URL=sqlite3://test.db
    ./juicefs format $META_URL myjfs --trash-days 0 --enable-acl
    ./juicefs mount -d $META_URL /jfs --enable-xattr
    ./juicefs dump $META_URL dump.json --fast
    ./juicefs load sqlite3://test2.db dump.json
    ./juicefs dump sqlite3://test2.db dump2.json --fast
    diff -ur dump.json dump2.json

Result:
--- dump.json	2024-03-18 15:30:08.348024722 +0800
+++ dump2.json	2024-03-18 15:30:08.348024722 +0800
@@ -24,7 +24,7 @@
     }
   },
   "Trash": {
-    "attr": {"inode":0,"type":"directory","mode":0,"uid":0,"gid":0,"atime":0,"mtime":0,"ctime":0,"nlink":1,"length":0},
+    "attr": {"inode":0,"type":"directory","mode":0,"uid":0,"gid":0,"atime":0,"mtime":0,"ctime":0,"nlink":2,"length":0},
     "entries": {
     }
   }

What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?
Environment:

JuiceFS version (use juicefs --version) or Hadoop Java SDK version:
Cloud provider or hardware configuration running JuiceFS:
OS (e.g cat /etc/os-release):
Kernel (e.g. uname -a):
Object storage (cloud provider and region, or self maintained):
Metadata engine info (version, cloud provider managed or self maintained):
Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
Others:

root@bench-01:~/juicefs# cat dump.json
{
"Setting": {
"Name": "myjfs",
"UUID": "2ac31bec-e004-4563-8436-e87a5ea34845",
"Storage": "file",
"Bucket": "/var/jfs/",
"BlockSize": 4096,
"Compression": "none",
"EncryptAlgo": "aes256gcm-rsa",
"TrashDays": 0,
"MetaVersion": 1,
"MinClientVersion": "1.2.0-A",
"DirStats": true,
"EnableACL": true
},
"Counters": {
"nextTrash": 0
},
"Sustained": [],
"DelFiles": [],
"FSTree": {
"attr": {"inode":0,"type":"directory","mode":511,"uid":0,"gid":0,"atime":1710747007,"mtime":1710747007,"ctime":1710747007,"atimensec":649659428,"mtimensec":649659428,"ctimensec":649659428,"nlink":2,"length":0},
"entries": {
}
},
"Trash": {
"attr": {"inode":0,"type":"directory","mode":0,"uid":0,"gid":0,"atime":0,"mtime":0,"ctime":0,"nlink":1,"length":0},
"entries": {
}
}
}


Move scp to standard transport

The PR is based on part of #1602.
#1602 is in two parts:-

Move SCP away from using struct SCP_CONNECTION as a connection type to the more standard struct trans transport type used everywhere else
Add the capability to use UNIX domain sockets for SCP

Both of these are necessary moving forwards to modernise SCP. This PR focusses on the first half only.
I've taken @jsorg71's original commits and rebased them. I've then completely removed the struct SCP_CONNECTION to avoid confusion, and to ensure that the transition to struct trans is complete. This has entailed updating the utilities.
There's still a lot to do for SCP. This is only a start.

Review comments addressed.

Thanks for this, it will be great to removing this unnecessary TCP listener.

There's 1 month until the next release. This is a breaking change so we need to triage if shipping this to August release or December release. Do you think 1 month is enough to test and debug this?

I should point out that a fair bit of this code isn't actually used at present.
The server-side SCP V0 code is used by xrdp and sesrun, and even they aren't using the SCP client-side library at the moment. The SCP V1 admin code is used by sesadmin, and the other SCP V1 code is used by sestest which isn't shipped by default, primarily as it isn't completed yet.
I've tested all the interfaces I can. xrdp/sesrun appear to work well, as does what there is of sesadmin. The sestest code run OK, but has other (existing) problems which I'm not planning on addressing right now.
In other words, I think an August release for this is realistic, but happy to discuss this further. I'm about for most of August apart from a few days from the 15th onwards.
If we need to bounce it until December that's fine, but I'd suggest a merge straight after v0.9.17 early in September if we do that. There are other changes to make in this area, but they're dependent on this one going in.

Then, I'm fine with shipping this to next August release.
Just to be sure, does this PR switches connection between xrdp and sesman from tcp to uds?

No it doesn't.
That's a subsequent part of Jay's original #1602. Because that's a significantly more user-visible change however, it will be good to get some more preparation done before we make that switch, so it's clearer what's going on.
In particular, at the moment, xrdp and sesrun don't use libscp to talk to sesman. This PR doesn't address that, but I've got another one planned that will.
After what we should be able to move to UDS with a relatively simple PR.

Okay, let's ship this to the next release.


Fixed tweening floors.

Sometimes "layer" component loads later and @layer?.name is undefined for the first frame and it's enough for tweening

Does adding the or isLand short circuit the if statement and thus prevent the tween?

it's not a short curcuit, isLand is the "floor thang" property by default. layer.name is another component and for floor thangs it's not required if isLand is checked. But maybe layer.name is 'ground' is using for other thangs so I haven't removed it.


issue with emulator

404 [conversationUpdate]
This is the issue I'm facing with I have already tried many but didnot find any result to overcome this

Hi @sailav, What was the endpoint you specified in the emulator? A typical localhost endpoint looks like: http://localhost:3978/api/messages. I've seen 404s returned when the path part of the bot endpoint is in some way incorrect. For example, if you only specified http://localhost:3978, the bot would return a 404.

Going ahead and closing this now. If you'd like further assistance, reply here and we'll reopen it.

@eanders-MS
[20:19:58]Emulator listening on http://localhost:36873
[20:19:59]Failed to spawn ngrokspawn UNKNOWN
[20:19:59]
[20:19:59]POST401directline.conversationUpdate
[20:19:59]
[20:19:59]POST401directline.conversationUpdate
[20:19:59]POST201directline.startConversation
[20:20:04]-> messagehi
[20:20:04]Cannot post activity. Unauthorized.
[20:20:04]POST401directline.postActivity
I also getting these error, I don['t know why.....

Either it should be ngrok settings issue or something you written in you
both backend
On Sun, 11 Aug, 2019, 8:23 PM Deepanshu Ruhela<EMAIL_ADDRESS>wrote:

@eanders-MS https://github.com/eanders-MS
[20:19:58]Emulator listening on http://localhost:36873
[20:19:59]Failed to spawn ngrokspawn UNKNOWN
[20:19:59]
[20:19:59]POST401directline.conversationUpdate
[20:19:59]
[20:19:59]POST401directline.conversationUpdate
[20:19:59]POST201directline.startConversation
[20:20:04]-> messagehi
[20:20:04]Cannot post activity. Unauthorized.
[20:20:04]POST401directline.postActivity
I also getting these error, I don['t know why.....
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/BotFramework-Emulator/issues/190?email_source=notifications&email_token=AG2MMZGBGZT4PC32ZUBROCLQEARXVA5CNFSM4DK7PORKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4BCL7A#issuecomment-520234492,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG2MMZCS2IBIFCSFLFD7OUTQEARXVANCNFSM4DK7PORA
.

@deepru2608 Regarding the ngrok issue, do you have the path to the ngrok executable configured in your emulator app settings?

Also, I can see that you are getting a 401 error at the start of the conversation. This usually means that when you connected to the bot, you did not enter the correct credentials in the Microsoft App ID & Microsoft App Password fields.

What you enter into these input fields must match the credentials in your bot code.
===
Also, please don't mention eanders-MS because he no longer maintains this project.

@tonyanziano
Thanks you so much for your help.
ngrok path is missing so i am getting that kind pf error.
and the reason of 401 error is bot emulator problem. i have update my bot than it's good to go.
and one more thing i am not using any kind of credential because i haven't deploy my bot.


Update _index.md

Make 100% clear, that according to the k8s networking model pods are supposed to get their own unique IP address. I always thought this was the case. But then faced debugging issue where some pods are being assigned the same IP addresses. Came here to review my understanding of it, and confirm this is expected behaviour or not. But it was not stated unequivocally. This change makes it so.
Refs #32360

/lgtm

@divya-mohan0209 @sgpinkus can we please recover this section https://github.com/kubernetes/website/pull/32363/files#r830935534 , the current documentation is wrong, there is no way to guarantee that hostNetwork pods and agents communicate without NAT https://docs.google.com/presentation/d/1pjptuK0IjqjDCao4sIUFUZ4qdpsq1o1elcHSeqKj9xM/edit#slide=id.gd9a836c4e3_0_39
/cc: @sftim

@aojea by recovering, do we mean reverting the change as it is right now?

Yep, we sure can @aojea although I reckon it will need to be done manually. Thank you for flagging this, I will open up an issue to target this file specifically.
Just to clarify, do we only need suggestions from comment 1 and comment 2 to make their way into this file to make it more technically accurate? Or would we need a complete re-sync of the file alongside the changes suggested by #28454 ?

do we only need suggestions from https://github.com/kubernetes/website/pull/32363#discussion_r1413838618 and https://github.com/kubernetes/website/pull/32363#discussion_r1413840655 to make their way into this file to make it more technically accurate

yes, no need to complete resync, you are better with the wording, I just want to give the same consistent message, I let you to decide how


"inFile" option?

It would be nice to be able to use the nice features of roku-deploy using a pre-packaged app, like:
roku-deploy -infile bin/app.zip -host <IP_ADDRESS> -password abc

Even using the internal JS API it isn't easy to achieve.

Related to (and will be addressed by ) #161


remove repeat load history_plugin

remove repeat load history_plugin from nodeos .
it have been loaded at eos/programs/nodeos/CMakeLists.txt.

If the repeated line you mentioned is https://github.com/EOSIO/eos/blob/master/programs/nodeos/CMakeLists.txt#L55, I think it's not for loading history_plugin in runtime, but it tries link nodeos to history_plugin during compilation. In that case, nodeos doesn't load history_plugin repeatedly.

It is actually registered via:
https://github.com/EOSIO/eos/blob/master/plugins/history_plugin/history_plugin.cpp#L17
You are correct that explicit register in main is not needed.

okay , misunderstand about that . but same result .
hah ! very nice .


ArtStorm.nft Project

Summary:
This issue delves into the successful execution of ArtStorm.nft, a collaborative project that empowered artists and revitalized urban spaces in Africa through creative street art and NFTs. Partnering with Eth Safari, Public Space Network, and About Art Africa, we achieved the following:

Transformed 20+ disused spaces in Nairobi and Kilifi, Kenya, into vibrant green art hubs, promoting environmental sustainability and community well-being.
Empowered 23 talented African artists with new creative opportunities and income through NFT sales on Foundation.app.
Created "African Streets of the Future," a captivating NFT collection showcasing visions of a greener, more vibrant Africa.
Generated media coverage and online engagement, raising awareness about the project's impact and the potential of art and technology for positive social change in Africa.

Key Achievements:

Artist Onboarding & Support: Successfully transitioned and onboarded 23 artists from traditional art practices to the NFT space, providing comprehensive guidance and technical support.
Collection Development: Curated a compelling "African Streets of the Future" collection featuring diverse artistic styles and insightful narratives about a transformed Africa.
NFT Launch & Sales: Executed a successful Foundation.app launch, generating [specify the amount or range] in NFT sales and providing artists with valuable earning opportunities.
Community Engagement: Fostered collaboration between artists, placemakers, and local communities, promoting ownership and a sense of shared responsibility for green spaces.

Evaluation & Learnings:

Artist Outreach: Expanding outreach strategies to include a wider range of artistic communities and geographic locations for future iterations.
NFT Education: Developing deeper educational resources to enhance artist confidence and understanding of the NFT landscape.
Community Participation: Exploring further avenues for increased community involvement in project decision-making and artistic curation.
Impact Measurement: Refining metrics and data collection methods to comprehensively track the project's social and environmental impact beyond financial outcomes.

Next Steps:

Sharing Data & Insights: Openly sharing project data, learnings, and best practices with the artistic community and related organizations.
Refining Future Iterations: Utilizing feedback and evaluations to continuously improve the ArtStorm.nft model for future implementations in other African cities.
Expanding Partnerships: Seeking collaborations with additional partners to amplify the project's reach and impact across the continent.

Call to Action:
We invite the community to actively participate in discussions, share feedback, and collaborate on building a greener, more artistic future for African cities. Join us in harnessing the power of art and technology for positive social change!
Please share your thoughts, suggestions, and questions in the comments below. Let's continue fostering creativity, building vibrant communities, and transforming urban spaces together!

Summary:
This issue delves into the successful execution of ArtStorm.nft, a collaborative project that empowered artists and revitalized urban spaces in Africa through creative street art and NFTs. Partnering with Eth Safari, Public Space Network, and About Art Africa, we achieved the following:

Transformed 20+ disused spaces in Nairobi and Kilifi, Kenya, into vibrant green art hubs, promoting environmental sustainability and community well-being.
Empowered 23 talented African artists with new creative opportunities and income through NFT sales on Foundation.app.
Created "African Streets of the Future," a captivating NFT collection showcasing visions of a greener, more vibrant Africa.
Generated media coverage and online engagement, raising awareness about the project's impact and the potential of art and technology for positive social change in Africa.

Key Achievements:

Artist Onboarding & Support: Successfully transitioned and onboarded 23 artists from traditional art practices to the NFT space, providing comprehensive guidance and technical support.
Collection Development: Curated a compelling "African Streets of the Future" collection featuring diverse artistic styles and insightful narratives about a transformed Africa.
NFT Launch & Sales: Executed a successful Foundation.app launch, generating [specify the amount or range] in NFT sales and providing artists with valuable earning opportunities.
Community Engagement: Fostered collaboration between artists, placemakers, and local communities, promoting ownership and a sense of shared responsibility for green spaces.

Evaluation & Learnings:

Artist Outreach: Expanding outreach strategies to include a wider range of artistic communities and geographic locations for future iterations.
NFT Education: Developing deeper educational resources to enhance artist confidence and understanding of the NFT landscape.
Community Participation: Exploring further avenues for increased community involvement in project decision-making and artistic curation.
Impact Measurement: Refining metrics and data collection methods to comprehensively track the project's social and environmental impact beyond financial outcomes.

Next Steps:

Sharing Data & Insights: Openly sharing project data, learnings, and best practices with the artistic community and related organizations.
Refining Future Iterations: Utilizing feedback and evaluations to continuously improve the ArtStorm.nft model for future implementations in other African cities.
Expanding Partnerships: Seeking collaborations with additional partners to amplify the project's reach and impact across the continent.

Call to Action:
We invite the community to actively participate in discussions, share feedback, and collaborate on building a greener, more artistic future for African cities. Join us in harnessing the power of art and technology for positive social change!
Please share your thoughts, suggestions, and questions in the comments below. Let's continue fostering creativity, building vibrant communities, and transforming urban spaces together!

Updates

Project Live link
Press Release

This will be review by the support team. Visit the live support page to initiate a chat with an agent for further assistance via live chat HERE  LIVE CHAT SUPPORT Regards


Certificate serial number reported in wrong format

When testing a prototype for a future tool (#103), I found that a self-signed certificate for a printer's web UI had a negative serial number. Looking closer, I found that all serial numbers that have been reported since v0.1.0 have been in an incorrect format.
From a README example:
Certificate 1 of 2 (leaf):
        Name: CN=www.google.com,O=Google LLC,L=Mountain View,ST=California,C=US
        SANs entries: [www.google.com]
        KeyID: 8E:A3:6C:47:12:A7:A:7:5B:94:51:D6:2A:3F:72:F9:35:6:45:2C
        Issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US
        IssuerKeyID: 98:D1:F8:6E:10:EB:CF:9B:EC:60:9F:18:90:1B:A0:EB:7D:9:FD:2B
        Serial:<PHONE_NUMBER>93767042001244177974291853363
        Expiration: 2020-09-09 14:31:22 +0000 UTC
        Status: [WARNING] 65d 3h remaining

Looking at the code, I find:
https://github.com/atc0005/check-cert/blob/6b1fad083faca657d60a7efcf29dde931f566c4b/internal/certs/certs.go#L339
https://github.com/atc0005/check-cert/blob/6b1fad083faca657d60a7efcf29dde931f566c4b/internal/certs/certs.go#L357
and according to the docs (https://golang.org/pkg/math/big/#Int.String), the Stringer interface:

String returns the decimal representation of x as generated by x.Text(10).

Which means that technically the output is the right serial number, it isn't in the hex format that everyone is used to working with.
Which means that this line in the README example (base 10):
        Serial:<PHONE_NUMBER>93767042001244177974291853363

should have been (base 16):
        Serial: FD6F3E2498C25B1D080000000047F033

Code to convert base 10 *bit.Int output to base 16
package main

import (
	"fmt"
	"math/big"
)

func main() {
	serialNumber := new(big.Int)
	serialNumber.SetString("336872288293767042001244177974291853363", 10)
	fmt.Printf("%v\n", serialNumber.Text(16))
}

Output:
fd6f3e2498c25b1d080000000047f033

I'm testing a few fixes, but I'm still running into a case where a self-signed printer certificate is resulting in a serial number that does not match what Google Chrome and Firefox provides.

Worth nothing that the last two lines give equivalent results:
package main

import (
	"fmt"
	"math/big"
)

func main() {
	serialNumber := new(big.Int)
	serialNumber.SetString("336872288293767042001244177974291853363", 10)
	fmt.Printf("%v\n", serialNumber.Text(16))
	fmt.Printf("%x\n", serialNumber)
}

I'm testing a few fixes, but I'm still running into a case where a self-signed printer certificate is resulting in a serial number that does not match what Google Chrome and Firefox provides.

Per Chromium (Ubuntu) or Firefox (Ubuntu):
D6:DA:0A:57:2A:1D:03:3C:8E:88:0B:B7:C5:F6:D1:DB:F0:F1:C8:E6
Per a current build of lscert:
-2925f5a8d5e2fcc37177f4483a092e240f0e371a

This is interesting:
$ echo "" | openssl s_client -connect <IP_ADDRESS>:443 2>&1 | openssl x509 -text | grep -A 1 'Serial Number'
        Serial Number:
             (Negative)29:25:f5:a8:d5:e2:fc:c3:71:77:f4:48:3a:09:2e:24:0f:0e:37:1a

Perhaps I'm not entirely off-base. Why are Chrome and Firefox giving different serial number results?

After some research, it appears that some systems generate negative serial numbers, even if this goes against later standards (RFC3280). Based on those standards, clients are supposed to be capable of handling situations where a negative serial number is returned. Presumably this is for backwards compatibility.
From the RFC:

<IP_ADDRESS>  Serial number
The serial number MUST be a positive integer assigned by the CA to
each certificate.  It MUST be unique for each certificate issued by a
given CA (i.e., the issuer name and serial number identify a unique
certificate).  CAs MUST force the serialNumber to be a non-negative
integer.
Housley, et. al.            Standards Track                    [Page 17]

RFC 3280        Internet X.509 Public Key Infrastructure      April 2002
Given the uniqueness requirements above, serial numbers can be
expected to contain long integers.  Certificate users MUST be able to
handle serialNumber values up to 20 octets.  Conformant CAs MUST NOT
use serialNumber values longer than 20 octets.
Note: Non-conforming CAs may issue certificates with serial numbers
that are negative, or zero.  Certificate users SHOULD be prepared to
gracefully handle such certificates.

At one point Go aggressively failed certs with negative serial numbers, but due to the prevalence of negative serial numbers in circulation, the team relented and removed the validity check that prevented interaction/use of certs with negative serial numbers.
Some sources I found in my research indicated that Mac OS X (2016) incorrectly displayed certs with serial numbers as positive numbers. Perhaps that is what Firefox and Chrome are doing?

This mozilla/tls-observatory GH issue notes the same problem I ran into yesterday while testing:

There is a bug in the certificate parser. We use
stored.Serial = strings.ToUpper(hex.EncodeToString(cert.SerialNumber.Bytes()))
where cert.SerialNumbercert.SerialNumber is a bigint and the Bytes() method "returns the absolute value of x as a big-endian byte slice."
Using fmt.Sprintf() is much simpler, I'll push a patch.

I also noted that the negative serial number was modified to drop the sign, which further confused me as I was getting different results from Go, OpenSSL and Firefox/Chrome.
I also noted that fmt.Sprintf() gave the same results as OpenSSL. Presumably OpenSSL is the correct reference to test our results against.


External command parameters borked by ValidCommandParameterNames

During experiments, snippet
docker run -id blabla

was changed to
docker run -Id

I presume that -Id is a learned CommandParameter and is stored in ValidCommandParameterNames table. However, maybe it shouldn't be applied for parameters of external commands? Not sure how to fix it though. We would need to detect whether we're inside powershell or an external command.
Slightly related https://github.com/DTW-DanWard/PowerShell-Beautifier/issues/41

Yep, you've hit upon some of the limitations with the current implementation.  Right now the replacement logic is not very smart - for example it can't easily tell PowerShell parameters versus those for external commands.  I do have a plan for fixing this but it involves a complete re-write and using abstract syntax trees.  I've started on this work but it is going to be a long process; I don't expect a beta version for many months.  The plus side: it will have some crazy awesome functionality (not just whitespace cleanup and simple text casing fixes), will be very configurable and will allow users to write their own rules.

Closing this for now; will be addressed (hopefully) in next-gen version.


Api call for multiple use cases

I have a viewcontroller that has multiple usecases, both of which make calls to an Api and does so on viewcontroller load. Currently, a operationqueue calls the usecases 1 at a time. How would I approach making these calls concurrent? I want both usecases to have their own thread to make their call on load.

OperationQueue is concurrent by default, so as long as you call func addOperation(_ block: @escaping () -> Void) to run your use case then all use cases should run concurrently (you might want to have a look at var maxConcurrentOperationCount: Int { get set } as well).

This tutorial might help as well.

Where would I call addOperation()

👍🏾


Integrate RASPA into workflow

Call RASPA after charge calculation finishes

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

7 of 7 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 46.043%

Totals

Change from base Build<PHONE_NUMBER>:
0.3%

Covered Lines:
4416

Relevant Lines:
9591

💛  - Coveralls


Update MainWindow.cs

use @echo off just like in https://github.com/Klocman/Bulk-Crap-Uninstaller/blob/f9ddbea5a3cd881a7233e61f50eb98d19596b3bd/source/BulkCrapUninstaller/CleanLogs.bat#L2 instead of cls

Thank you 👍

Thank you 👍

np. Don't thank me for such a small change. I'd rather want to add more infos to the batch script, something along the lines of
Do you want the installers to run simulataneously? Y/N
if Y then
    start "" "<uninstaller" <args>
else
   start /wait "" "<uninstaller" <args>

and then have a line between each giving progress infos:
Starting uninstaller for <name> <version> [ <i> / <count> ]


Added user_get_auth_ticket_for_web_api

Excellent! Thank you for the contribution. Could you please also run generatedocs.py to update the .api_script and related documentation files?


scp fails due to shell response

scp gives an error: "Received message too long"
The easiest solution is adding a single line to the top of ~/.bashrc:
[[ $- == *i* ]] || return

Ha, someone's fixed it since I installed this :) I'll close


Prevent panic in ResponseError.Error()

If RawResponse is nil (unit test scenarios), print "nil RawResponse" instead of panicing.
Fixes https://github.com/Azure/azure-sdk-for-go/issues/21838

Customers should never be making programmatic choices based on the error message string as they are subject to change. Or did I misunderstand the question?

Customers should never be making programmatic choices based on the error message string as they are subject to change. Or did I misunderstand the question?

Before this change the app terminated with a panic.  now it will return an error.  The customer will need to do something... We should guide them on how to handle this newfound freedom.

SDKs only return an *azcore.ResponseError when the service returns a 400 (and possibly other) HTTP status code, which means we have a *http.Response. If a SDK method were ever to return an *azcore.ResponseError with a nil RawResponse field it means we have a bug either in the SDK or azcore.


💥 libhal 5.0.0

Closing this for now. Will come back to this to take parts from it and add them to libhal 4.x.y. Once all of the parts that do not conflict with 5.0.0 have been moved into 4.x.y, then this PR will come back to make the breaking API/ABI changes.


[SPARK-12177][Streaming][Kafka] Update KafkaDStreams to new Kafka 0.10 Consumer API

What changes were proposed in this pull request?
New Kafka consumer api for the released 0.10 version of Kafka
How was this patch tested?
Unit tests, manual tests

Test build #53680 has started for PR 11863 at commit 546246e.

Test build #53680 has finished for PR 11863 at commit 546246e.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53680/
Test FAILed.

Test build #53682 has started for PR 11863 at commit 477055c.

Test build #53682 has finished for PR 11863 at commit 477055c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53682/
Test FAILed.

Merged build finished. Test FAILed.

Test build #53686 has started for PR 11863 at commit ba41956.

Test build #55252 has started for PR 11863 at commit e559183.

Test build #55252 has finished for PR 11863 at commit e559183.

This patch fails MiMa tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55252/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55363/
Test FAILed.

Merged build finished. Test FAILed.

Thanks for this, @koeninger! This is looking really good. I have made some comments inline.
I noticed that there wasn't a new example, a new KafkaUtils, python support or documentation. So, we should take care of those too.
Also, I'd personally prefer a different name than kafka-beta, because the new consumer API will become GA some day and I would rather not have us rename the project then and break everyone. I would personally prefer calling it some thing like kafka-newapi (or something else, it's not that important). We can doc that this subproject is beta quality but as time goes and the subproject and kafka's new consumer API matures, we don't have to rename, just update the docs.
And, I would definitely like to help out with the work you have started here. Whether it's implementing some of the comments (if you agree with them), or the left over things that I listed out earlier. Let me know what you think.
Thanks again for working on this!

As far as the name, I don't really care.  If you name it beta, one day it
will be GA.  If you name it newapi, one day it will be the old api.
Stuff's going to change either way.  Plenty of people won't read the docs,
but having beta in the name should make it realllly obvious what the state
of the code is.
As far as python support, I'm not really a python user, so probably not the
best person to ask as far as making a nice api for python.  My bet is that
adequately mapping all of the possible things that can be done to set up
the underlying consumer is going to be quite hard.
As far as examples, I'd be fine with just copying over the stuff from
https://github.com/koeninger/kafka-exactly-once/tree/kafka-0.9
As far as KafkaUtils, we can add it back in to avoid rocking the boat, but
honestly I don't think we get much benefit from it.  In the old  consumer
it ended up being a dumping ground for random stuff from 3 or 4 different
classes.
On Fri, Apr 15, 2016 at 7:25 PM, Mark Grover<EMAIL_ADDRESS>wrote:

Thanks for this, @koeninger https://github.com/koeninger! This is
looking really good. I have made some comments inline.
I noticed that there wasn't a new example, a new KafkaUtils, python
support or documentation. So, we should take care of those too.
Also, I'd personally prefer a different name than kafka-beta, because the
new consumer API will become GA some day and I would rather not have us
rename the project then and break everyone. I would personally prefer
calling it some thing like kafka-newapi (or something else, it's not that
important). We can doc that this subproject is beta quality but as time
goes and the subproject and kafka's new consumer API matures, we don't have
to rename, just update the docs.
And, I would definitely like to help out with the work you have started
here. Whether it's implementing some of the comments (if you agree with
them), or the left over things that I listed out earlier. Let me know what
you think.
Thanks again for working on this!
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-210696664

Test build #56094 has started for PR 11863 at commit ec095fb.

Test build #56094 has finished for PR 11863 at commit ec095fb.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56094/
Test FAILed.

Overall, I agree with the approach. I'll take a look at kafka-exactly-once repo, I don't think it will hurt to add more tests.
As far as KafkaUtils go, I am ok with getting rid of it for Scala/Java users but there are a bunch of python utility functions, the equivalent of which will likely be needed if/when we do python support. But, we could just add it later if users asks for it, or we find it's the best place for python utility functions to go.

Test build #57996 has started for PR 11863 at commit 544bf88.

Test build #57996 has finished for PR 11863 at commit 544bf88.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57996/
Test FAILed.

Test build #59496 has started for PR 11863 at commit a26160f.

Test build #59496 has finished for PR 11863 at commit a26160f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59496/
Test FAILed.

Test build #59499 has started for PR 11863 at commit 7cf5495.

Test build #59499 has finished for PR 11863 at commit 7cf5495.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59499/
Test FAILed.

Test build #60521 has started for PR 11863 at commit 82067c0.

Test build #60521 has finished for PR 11863 at commit 82067c0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60521/
Test FAILed.

Test build #60550 has started for PR 11863 at commit c715872.

Test build #60550 has finished for PR 11863 at commit c715872.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60550/
Test FAILed.

Jenkins, retest this please

Test build #60557 has started for PR 11863 at commit c715872.

Test build #60557 has finished for PR 11863 at commit c715872.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60557/
Test FAILed.

Test build #60572 has started for PR 11863 at commit 3a95969.

Test build #60575 has started for PR 11863 at commit 9dd7bce.

Test build #60572 has finished for PR 11863 at commit 3a95969.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60572/
Test FAILed.

Test build #60575 has finished for PR 11863 at commit 9dd7bce.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60575/
Test FAILed.

Test build #60606 has started for PR 11863 at commit b404304.

Test build #60606 has finished for PR 11863 at commit b404304.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60606/
Test FAILed.

Jenkins, retest this please
On Jun 15, 2016 8:43 PM, "UCB AMPLab"<EMAIL_ADDRESS>wrote:

Merged build finished. Test FAILed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-226367273, or mute
the thread
https://github.com/notifications/unsubscribe/AAGAB_z_ud5CYHtZK84TlywHg5QWqmXuks5qMKpWgaJpZM4H1Pg1
.

Jenkins, retest this please

Test build #60622 has started for PR 11863 at commit b404304.

Test build #60622 has finished for PR 11863 at commit b404304.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60622/
Test PASSed.

@rxin I'm pretty confident that fix addressed the test issue.  It's passed twice now, the prior failure was unrelated.

@koeninger I am reading the whole code and understand how it works. I am just marking some style stuff along the way, dont worry about them for now. Let me understand the whole thing and then i will have more important comments.

@koeninger Okay, I think I understand the code at a high level. And I have a number of architecture and API concerns. Let me start with the highest level architectural concern to keep the discussion focused.

This code expects the user to figure out the prefered hosts for each Kafka partition, and then just keeps setting that as the preferred location for RDD partition. So if the user sets that to the default Map.empty, then what happens? Will each RDD partition get randomly assigned to the executors? Then every executor will have one cached consumer for each partition. Would that not lead to scalability problems for the Kafka cluster?

i think the right way here is to automatically record the executors that read each topic+partition in the previous batch, and use those executors as the preferred ones in the next batch. And ideally, periodically adjust if the executor-to-partition assignment is too unbalanced. The last one may be done later, but we should at least do the first one. In other words, "consistent" scheduling should be out of the box.

Next, let me explain the API concerns. Note that this is the 2.0 release, and we are extremely conservative about the new APIs that we are exposing, as we will ideally have to support them for the 2.x.y line. So we have to be absolutely sure about the public API we expose, and its better not to expose it if we are not sure about being able to support it for the next few years (even if 3rd party library breaks compatibility and all). Also we have learnt a bunch of things from mistakes in the past - mistakes like exposing 3rd-party library classes in the Spark API which is forcing us to make kafka-0-8 and kafka-0-10. All that said, I find the current DirectKafkaInputStream API very weird. It took me and @zsxwing couple of hours to really understand how to use it. Here are the concerns.

The driverConsumer function is the most confusing. Looking at it, its not clear how i create the function such that I want to start from a particular offset or the latest offset. Why not reciprocate the KafkaUtils.createDirectStream() API which takes fromOffsets? I believe that is a much simpler API for the user as the user does not have to learn how to write the driverConsumer such that it starts from a particular offset? And that is a much narrow API+functionality that can be supported no matter how things change in Kafka.

There does not seem to be real need for the having public vars preferBrokers and preferConsistent. All the user should have to say is "optionally" here is the preferred locations. If the the preferred location is empty, then the system ensures consistent scheduling. That's all.

We MUST NOT expose any Kafka classes in the API. That includes TopicAndPartition. I would create a mirror class for that.

DirectKafkaInputStream and KafkaRDD should not be public at all. They were not public in 0.8, they should not be public in 0.10 either.

Regarding the possible arguments to preferredHosts, I'm pretty sure you're
misunderstanding what happens.
There are 3 uses cases here:

I don't care where things run.  I pass preferConsistent, and the
definition of KafkaRDD.getPreferredLocations does the right thing.
I know my executors are on the same nodes as my brokers, and I want
locality.  I pass preferBrokers, and getPreferredHosts figures out the
brokers for me.
I want to specify the mapping, because of skewed topics or whatever.  I
pass my own map.

If you think this would be less confusing if the static constants didn't
exist and the docs said "pass null" instead of "pass preferBrokers" and
"pass an empty map" instead of "pass preferConsistent", sure, we can do
that. If we want a full fledged interface here, ok, I think it's overkill,
but I'll do it.  But your description of what the code is currently doing
is not accurate, and your recommendation does not meet the use cases.
Regarding DirectKafkaInputDStream and KafkaRDD being visible vs being
hidden in KafkaUtils, that's fine for consistency sake, no prob.
Regarding your other API concerns... this has been discussed pretty heavily
in the JIRA for months.  Have you read any of that discussion?
Bottom line, I guarantee you that I cannot do my job with this connector
unless I have unfettered ability to create the Consumer on the driver side.
If we want to provide convenience methods to create a function setting up a
Consumer given just a list of topics or a list of offsets, great, we can do
that.  But without being able to arbitrarily configure the consumer,
getting things like dynamic topic subscriptions working without auto offset
reset is just not going to be possible... to say nothing of allowing people
to create their own custom offset generation policies without having to
recompile spark.
The problem here is not that we made too much stuff public in 0.8.  The
problem here is that the semantics and abilities of the underlying consumer
have changed - things like messageHandler in 0.8 flat out wouldn't work any
more.  Hiding more of the Kafka classes would not have helped, at all.
Keep in mind that we are not requiring a concrete Kafka class, we're
requiring an implementation of an interface (Consumer, not KafkaConsumer).
If the Kafka project drastically adds to that interface for 0.11, this
current code isn't going to have to change much (if at all).
If instead, we wrap everything with Spark reimplementations of all the
Kafka interfaces, it's going to have to change quite a bit, regardless.
On Wed, Jun 22, 2016 at 9:01 PM, Tathagata Das<EMAIL_ADDRESS>wrote:

@koeninger https://github.com/koeninger Okay, I think I understand the
code at a high level. And I have a number of architecture and API concerns.
Let me start with the highest level architectural concern to keep the
discussion focused.

This code expects the user to figure out the prefered hosts for each
Kafka partition, and then just keeps setting that as the preferred location
for RDD partition. So if the user sets that to the default Map.empty, then
what happens? Will each RDD partition get randomly assigned to the
executors? Then every executor will have one cached consumer for each
partition. Would that not lead to scalability problems for the Kafka
cluster?
i think the right way here is to automatically record the executors
that read each topic+partition in the previous batch, and use those
executors as the preferred ones in the next batch. And ideally,
periodically adjust if the executor-to-partition assignment is too
unbalanced. The last one may be done later, but we should at least do the
first one. In other words, "consistent" scheduling should be out of the box.
Next, let me explain the API concerns. Note that this is the 2.0 release,
and we are extremely conservative about the new APIs that we are exposing,
as we will ideally have to support them for the 2.x.y line. So we have to
be absolutely sure about the public API we expose, and its better not to
expose it if we are not sure about being able to support it for the next
few years (even if 3rd party library breaks compatibility and all). Also we
have learnt a bunch of things from mistakes in the past - mistakes like
exposing 3rd-party library classes in the Spark API which is forcing us to
make kafka-0-8 and kafka-0-10. All that said, I find the current
DirectKafkaInputStream API very weird. It took me and @zsxwing
https://github.com/zsxwing couple of hours to really understand how to
use it. Here are the concerns.

The driverConsumer function is the most confusing. Looking at it, its
not clear how i create the function such that I want to start from a
particular offset or the latest offset. Why not reciprocate the
KafkaUtils.createDirectStream() API which takes fromOffsets? I believe that
is a much simpler API for the user as the user does not have to learn how
to write the driverConsumer such that it starts from a particular offset?
And that is a much narrow API+functionality that can be supported no matter
how things change in Kafka.
There does not seem to be real need for the having public vars
preferBrokers and preferConsistent. All the user should have to say is
"optionally" here is the preferred locations. If the the preferred location
is empty, then the system ensures consistent scheduling. That's all.
We MUST NOT expose any Kafka classes in the API. That includes
TopicAndPartition. I would create a mirror class for that.
DirectKafkaInputStream and KafkaRDD should not be public at all. They
were not public in 0.8, they should not be public in 0.10 either.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-227929834, or mute
the thread
https://github.com/notifications/unsubscribe/AAGAByW_63A4018pBuCjg49Fq3-ClpGLks5qOekFgaJpZM4H1Pg1
.

I didnt quite get it when you meant "But your description of what the code is currently doing
is not accurate, and your recommendation does not meet the use cases." I just collapsed the three cases into two - when the user has NO PREFERENCES (the system SHOULD figure out how to schedule partitions on the same executors consistently), and SOME PREFERENCES (because of co-located brokers, or skew, or whatever). Why doesnt this recommendation meet the criteria?

I agree with the argument that there are whole lot of stuff you cannot do without exposing a () => Consumer function. Buts thats where the question of API stability comes in. At this late stage of 2.0 release, I would much rather provide simpler API for simpler usecases than we know will not break, rather than an API that supports everything is more prone to breaking if Kafka breaks API. We can always start simple and then add more advanced interfaces in the future.

Wrapping things up with extra Spark classes and interfaces is a cost we have to pay in order to prevent API breaking in the future. It is an investment we are undertaking in every part of Spark - SparkSession (using a builder pattern, instead of exposing constructor), SQL Data sources (never expose any 3rd party classes), etc. Its hard-learnt lesson.

When the user has no preferences, the system already does figure out
preferred locations, and not in a random way as you claimed.

So lets talk concretely, not hypothetically.  If we publish an api
where the constructor takes
() => Connector,
and we provide two simple ways for users to get an instance of that type,
e.g.
constructorFactory(listOfTopics)
and
constructorFactory(fromOffsets)

What is actually going to break when the Kafka project adds a new
subscribeAccordingToThePhaseOfTheMoon(moons) method to Consumer?  The
people using our simple factories go on about their business. The people
who are creating a consumer themselves can use the phase of the moon if
they want to, with a pretty minimal amount of change.
Non-hypothetically, the new Consumer already has a method for dynamic topic
subscription, which addresses some long-standing issues with the way the
0.8 consumer works.  Cutting people off of this because you're afraid of
something breaking makes no sense.  If people want to use something they
know is stable, with exactly the same features as the 0.8 connector....
they can still use the 0.8 connector with 0.10 brokers.

Again, concretely not hypothetically.

You're saying if only we had e.g. introduced a SparkWrappedMessage, and
made the 0.8 consumer messageHandler be
SparkWrappedMessage => R
instead of using the kafka class
MessageAndMetadata => R
all of this api change wouldn't have been necessary.
This is demonstrably false.  It would not have prevented api change.  The
behavior of the underlying consumer changed.  It changed in such a way
that we no longer have individual access to a message as its being
deserialized, because the consumer pre-fetches messages in blocks every
time it finishes a poll.  No amount of wrapping and hiding changes that.
I understand you've been burned on e.g. leaking classes from a myriad of
3rd party dependencies in core spark.  But the very purpose of this
standalone jar is to connect to kafka... the behavior allowed by the kafka
classes isn't incidental leakage, it's the whole point.
From my point of view, your stated goal is to minimize change.
My stated goal is to make sure people can use Kafka and Spark to get their
jobs done.
I'm demonstrably willing to do the maintenance work to make this happen,
even if things unavoidably change.  So are the other people who have worked
on this ticket since December of last year.
On Thu, Jun 23, 2016 at 12:39 PM, Tathagata Das<EMAIL_ADDRESS>wrote:

I didnt quite get it when you meant "But your description of what the
code is currently doing
is not accurate, and your recommendation does not meet the use cases."
I just collapsed the three cases into two - when the user has NO
PREFERENCES (the system SHOULD figure out how to schedule partitions on the
same executors consistently), and SOME PREFERENCES (because of co-located
brokers, or skew, or whatever). Why doesnt this recommendation meet the
criteria?
2.
I agree with the argument that there are whole lot of stuff you cannot
do without exposing a () => Consumer function. Buts thats where the
question of API stability comes in. At this late stage of 2.0 release, I
would much rather provide simpler API for simpler usecases than we know
will not break, rather than an API that supports everything is more prone
to breaking if Kafka breaks API. We can always start simple and then add
more advanced interfaces in the future.
3.
Wrapping things up with extra Spark classes and interfaces is a cost
we have to pay in order to prevent API breaking in the future. It is an
investment we are undertaking in every part of Spark - SparkSession (using
a builder pattern, instead of exposing constructor), SQL Data sources
(never expose any 3rd party classes), etc. Its hard-learnt lesson.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-228124879, or mute
the thread
https://github.com/notifications/unsubscribe/AAGAB5oB1G2I12GLilSrqXzd0DZnd6emks5qOsTBgaJpZM4H1Pg1
.

1+2. Architecture
I had to double check the code to understand the use of sorted list of executors to get consistent topic-to-preferred locations. PLEASE WRITE MORE SCALA DOCS on how the logic in the method work and what is returned! Otherwise its really tedious to understand all the nuances of the logic (in this case, when preferred location). Also, this does not completely eliminate my original concern because preferred location DOES NOT guarantee that the task will be scheduled on that executor. So tasks could get scheduled on other executors with some probability (especially if there is contention in the cluster, etc.) and small chance that multiple executors will have a cached consumer for the same Kafka partition. But this chance is small, and we can fix it later if this indeed becomes a problem.
3+4. API
My concern was with the Consumer class itself, it they change packaging and all. However, my colleagues convinced me that since this is a narrowly package focused on kafka-0-10 so if it breaks we can always release kafka-x-y. So I will concede on that. However, I still think we should provide simpler API for common use cases, and similar to the older KafkaUtils so that people trying migrate know what to look for. Here is my proposed API for direct stream and we can discuss feasibilty/pros/cons on this). This assumes that the common case is a fixed set of topics, user wants to start from earliest/latest/specific offsets.
object KafkaUtils {
  // Simpler API for common use cases
  def createDirectStream[K: ClassTag, V: ClassTag](
      ssc: StreamingContext,
      topicAndPosition: TopicPartitionOffset,    // defines the topics and starting offset, see classes below
      kafkaParams: Map[String, String],
      preferredHosts: Map[KafkaTopicPartition, String] = Map.empty): InputDStream[(K, V)] = ???

  // Advanced API for everything else
  def createDirectStream[K: ClassTag, V: ClassTag](
      ssc: StreamingContext,
      driverConsumer: () => Consumer[K, V],
      executorKafkaParams: ju.Map[String, Object],
      preferredHosts: ju.Map[TopicPartition, String] // this at last as this is least used, will be mostly empty
    ): InputDStream[(K, V)] = ???
}

/** Helper class to define the topics to subscribe and starting offsets. This should be JavaAPI */
object TopicPartitionOffsets {
    def latestOffsets(topics: Set[String]) = Latest(topics)
    def earliestOffsets(topics: Set[String) = Earliest(topics)
    def customOffsets(startingOffsets: Map[KafkaTopicPartition, Long]) = Custom(startingOffsets)
}

sealed abstract class TopicPartitionOffsets

Private[kafka] case class Latest(topics: Set[String]) extends TopicPartitionOffsets
Private[kafka] case class Earliest(topics: Set[String) extends TopicPartitionOffsets
Private[kafka] case class Custom(startingOffsets: Map[KafkaTopicPartition, Long])
extends TopicPartitionOffsets

The reason I am proposing TopicPartitionOffsets is to eliminate one case of overloading (see the two KafkaUtils.createDirectStream for scala in 0.8) AND also eliminate the weirdness of reusing kafka param "auto.offset.reset" for the purpose specifying latest/earliest. The API for Java createDirectStream would be similar. For RDD, KafkaUtils.createRDD would be similar to how it is currently in the KafkaRDD object.
Summary
To summarize in terms of API here are proposed changes

Add KafkaUtils with the above proposed API
Remove the DirectKafkaInputDStream and KafkaRDD objects. There is no need for them after KafkaUtils.

What do you think?

@koeninger Ping!

Sorry for the delayed reply, I had travel plans that had to be canceled due
to a family emergency (everyone's mostly ok).
1 + 2, I understand that preferred locations is not a guarantee.  Caching
should be a performance issue, not a correctness issue.  It's limited in
size so executors on the "wrong" hosts should eventually get phased out if
space is an issue.  I can add more comments as to the intention.
3 + 4, I agree that there should be an easy way for people to just specify
a list of topics without knowing about how Consumer works.  I agree that if
the convenience constructors are in KafkaUtils, the methods in the
companion objects for DirectKafkaInputDStream / KafkaRDD aren't necessary.
We can work out the specifics of the easy vs advanced api, but as long as
there's a way to get access to all of the Consumer behavior for advanced
users, I'm on board.    That's currently
ConsumerRecord[K,V].  If you need it to be a wrapped class that's fine as
long as it has the same fields, but that's another object instantiation for
each message.
TopicPartitionOffsets:  I agree with you that reducing overloads would be
good.  I agree with you that the old way we were using auto.offset.reset
was a little weird, because the simple consumer doesn't actually read that
parameter.  In this case however, the new consumer does read that parameter
and does use it to determine initial starting point (in the absence of a
specific seek to a specific offset).  Having more than one way to specify
the same thing is probably going to be more confusing, not less confusing.
I can probably come up with something that's a similar simple API, but it
may not look exactly like that.
So to summarize

I'll start tweaking things
Let me know if you think a wrapped class for ConsumerRecord is worth the
per-message overhead
Let me know if you're 100% attached to TopicPartitionOffsets

On Fri, Jun 24, 2016 at 12:47 PM, Tathagata Das<EMAIL_ADDRESS>wrote:

@koeninger https://github.com/koeninger Ping!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-228413198, or mute
the thread
https://github.com/notifications/unsubscribe/AAGAByQTQRImbjHpr4KDulcvYRkWN7Hyks5qPBg7gaJpZM4H1Pg1
.

Test build #61252 has started for PR 11863 at commit 0f15bd1.

Test build #61252 has finished for PR 11863 at commit 0f15bd1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61252/
Test FAILed.

Merged build finished. Test FAILed.

Test build #61259 has started for PR 11863 at commit 0610e9b.

Test build #61259 has finished for PR 11863 at commit 0610e9b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait CanCommitOffsets

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61259/
Test FAILed.

Test build #61272 has started for PR 11863 at commit 83c9c5b.

Moved the prefered location and consumer creation strategies to an explicit interface, let me know if that's more usable from your point of view.
Scala api looks like
KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
For starting from given offsets, the subscribe line would look like
FromOffsets[String, String](offsets,
Subscribe[String, String](topics, kafkaParams))
so that starting offsets can be used with arbitrary other setup strategies.
The ConsumerStrategy interface is public so that people can do arbitrary setup if need be.
Java api looks like
JavaInputDStream<ConsumerRecord<String, String>> istream2 = KafkaUtils.createDirectStream(
ssc,
String.class,
String.class,
PreferConsistent.create(),
Subscribe.create(String.class, String.class, Arrays.asList(topic2), kafkaParams2)
);

Test build #61272 has finished for PR 11863 at commit 83c9c5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait ConsumerStrategy[K, V]
case class Subscribe[K: ClassTag, V: ClassTag](
case class Assign[K: ClassTag, V: ClassTag](
case class FromOffsets[K: ClassTag, V: ClassTag](

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61272/
Test PASSed.

Merged build finished. Test PASSed.

I took a quick glance, and I think the following.

I like the LocationStrategy. But needs Java tests for each of the strategies.
Regarding ConsumerStrategy, I think I am trying to understand it a little more. And I am a little confused on what happens in the Subscribe strategy when the subscribed partitions change. I have left inline comments in the code.
Would nice to have LocationStrategy after ConsumerStrtategy in the parameter list, as the latter is more important than the former (location strategy can have preferConsistent as default).

Regarding your general comments
1 + 2 sure, will add

The reason ConsumerStrategy is last is because it's much more natural for inline one-off subclasses of it to customize setup.  There are only 3 arguments for the constructor at this point (ssc, locationStrategy, consumerStrategy) so I don't think requiring an explicit location strategy is much of a burden, and it makes sure people who were using brokers with the old api know what's going on.

4 a sure
b Yeah, I can test those.  Do you think there's a need to test things with implict args (e.g. classtags) from java, because those are super awkward.  Seems unlikely to be used when there's a version with explicit equivalent args.

Okay, no strong feelings on this one.
4b. I dont think there is a need for Java tests for methods which need class tags. Those are probably least likely to be used.

And I think you missed answering one of the comments in the code. Let me ask it once again. Can you explain what happens in the subscribe case when the consumer gets subscribed to a new partition? Which offset does it start from?

I was going to leave some code comments once I'm not at work & have time to make another patch, but the quick answer is that should be determined in the consumer by auto.offset.reset

what happens in the case where a partition was being processed by some other application, then gets assigned to this direct kafka stream. will it pick up the offsets that was last consumed by the other application (assuming the application committed correctly)?

If they're using the same group id, that's at least plausible, but it's in
general not a good idea to run two different applications with the same
group id at the same time.
On Mon, Jun 27, 2016 at 4:02 PM, Tathagata Das<EMAIL_ADDRESS>wrote:

what happens in the case where a partition was being processed by some
other application, then gets assigned to this direct kafka stream. will it
pick up the offsets that was last consumed by the other application
(assuming the application committed correctly)?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-228874354, or mute
the thread
https://github.com/notifications/unsubscribe/AAGAB7e6tcuh6KHkaHulOjkIyssRqLjIks5qQDqBgaJpZM4H1Pg1
.

Yeah, I guessed that its not a good idea, but I know some people do that. So I wanted to understand what actually would happen.

I'm already logging at error level a message about serializability / suggestion to use map first before persist, which seems like the most common case that will come up.  I can do something similar for .collect and .coalesce, anything else you can think of?

Test build #61359 has started for PR 11863 at commit db95290.

Test build #61359 has finished for PR 11863 at commit db95290.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61359/
Test PASSed.

Test build #61436 has started for PR 11863 at commit b1eec57.

Test build #61436 has finished for PR 11863 at commit b1eec57.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61436/
Test PASSed.

Merged build finished. Test PASSed.

@koeninger Sorry, I did not get a chance to review your changes because I was too busy with the programming guide PR (as you probably have guessed). I will jump back into your PR once that is merged, which will be soon. So tomorrow I will do another pass, and hopefully that will be final.

Test build #61495 has started for PR 11863 at commit 31502d9.

Finished my round of reviewing. Some some nits and one question about commitAsync left.

Test build #61506 has started for PR 11863 at commit f863369.

@zsxwing Thanks for the fixes

Overall, this is looking good. Two high level points.

Now we have two subprojects both creating org.apache.spark.streaming.kafka.KafkaUtils. I think this is going to cause problems downstream in the docs and stuff. Also it will be hard for user to disambiguate whats being used. So I propose changing the package name for the new one to o.a.s.streaming.kafka010. All the new classes will be in that package.

Don't need CanCommitOffsets.

We need a whole lot of new docs, especially updates in the streaming kafka integration guide. That will be a different PR. Could you start working on that?

You do need CanCommitOffsets because DirectKafkaInputDstream is now
private, so otherwise you have nothing to cast to to access that method.
On Jun 29, 2016 7:11 PM, "Tathagata Das"<EMAIL_ADDRESS>wrote:

Overall, this is looking good. Two high level points.

Now we have two subprojects both creating
org.apache.spark.streaming.kafka.KafkaUtils. I think this is going to cause
problems downstream in the docs and stuff. Also it will be hard for user to
disambiguate whats being used. So I propose changing the package name for
the new one to o.a.s.streaming.kafka010. All the new classes will be
in that package.
2.
Don't need CanCommitOffsets.
3.
We need a whole lot of new docs, especially updates in the streaming
kafka integration guide. That will be a different PR. Could you start
working on that?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-229525922, or mute
the thread
https://github.com/notifications/unsubscribe/AAGABz0T__MSSwsM6znRLSwfh8UwC1Kcks5qQwmqgaJpZM4H1Pg1
.

Test build #61495 has finished for PR 11863 at commit 31502d9.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61495/
Test FAILed.

Merged build finished. Test FAILed.

Aah, right. My bad. In that case, there arent major issues as far as i can see, let me merge this, and test how the docs look like. I am pretty sure its going to cause trouble with two KafkaUtils. And in that case I will handle the package renaming.

Well.. after the tests pass.

Test build #3150 has started for PR 11863 at commit f863369.

Test build #3151 has started for PR 11863 at commit f863369.

I'll do the scaladoc fix and the package rename.  I think the package rename is fine even if it did work with docs, just to disambiguate things.
Will start a separate ticket for documentation updates.

sounds good. thanks!

Test build #61513 has started for PR 11863 at commit cffb0e0.

Test build #61506 has finished for PR 11863 at commit f863369.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61506/
Test FAILed.

Test build #3151 has finished for PR 11863 at commit f863369.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #3150 has finished for PR 11863 at commit f863369.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #61513 has finished for PR 11863 at commit cffb0e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61513/
Test PASSed.

Merged build finished. Test PASSed.

LGTM. Merging this to master and 2.0. Thank you very much @koeninger for this awesome effort. :)

I played around with the API and I found a few issues

I mentioned above, case classes lead to problems in the public API. The API could be simpler, and same for both Java and Scala users (dont like the apply vs create).

The wrapping between java and scala maps were sometimes leading to the map not being serializable, and thus causing serialization issues in Subscribe and Assign strategies when checkpointing.

ConsumerStrategy class is an interface making it hard to add methods later without breaking compatibility.

I have opened a new PR to address them - please take a look - #13996

hey，I have an question about the setting "auto.commit.enable"， It could be changed ????Because I wanna save the offsets information to zookeeper cluster.

You won't get any reasonable semantics out of auto commit, because it will
commit on the driver without regard to what the executors have done.
On Aug 2, 2017 21:46, "Wallace Huang"<EMAIL_ADDRESS>wrote:

hey，I have an question about the setting "auto.commit.enable"， It could be
changed ????Because I wanna save the offsets information to zookeeper
cluster.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11863#issuecomment-319852845, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGAByVZ2QEf_o627Z7BKdRBuCtmza-Nks5sUTSTgaJpZM4H1Pg1
.

I'm wondering that why the setting "enable.auto.commit" existed, but it was set to false by default and I could't modify it . Anyway,  how do I use it ?


Regarding MAKE file of CUDA

When I m trying to rum make file for cuda it give me error . How to solve it
please help me

sandip@sandip-desktop:~/NVIDIA_GPU_Computing_SDK/C$ make
make[1]: Entering directory `/home/sandip/NVIDIA_GPU_Computing_SDK/C/common'
mkdir: cannot create directory `obj': Permission denied
make[1]: *** [makedirectories] Error 1
make[1]: Leaving directory `/home/sandip/NVIDIA_GPU_Computing_SDK/C/common'
make: *** [lib/libcutil.so] Error 2

Please suggest me some changes so that i can able to solve the problem

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 12 Feb 2012 at 11:33

Original comment by<EMAIL_ADDRESS>on 29 Feb 2012 at 11:47

Changed state: Invalid
Removed labels: Type-Defect, Priority-Medium

It appears you're trying to build the SDK. This has nothing to do with CUDPP.

Original comment by<EMAIL_ADDRESS>on 12 Feb 2012 at 3:47


Upgrade default Java version to 8

Is your feature request related to a problem? Please describe.
Over the past year we've had to limit Circus Train's usage of certain third party libraries which no longer support Java 7. This is becoming a maintenance pain and also prevents the usage of more advanced Java features in the code base. Java 7 is EOL so we shouldn't be using it in any production environments any more.
Describe the solution you'd like
JDK version in pom ugpraded to Java 8, Travis and other touch points upgraded to use Java 8.
Describe alternatives you've considered
N/A
Additional context
This proposal was sent to the mailing list on 2019-10 (https://groups.google.com/forum/#!topic/circus-train-user/40E_c_t0QLs) and nobody objected.

Fixed by https://github.com/HotelsDotCom/circus-train/pull/171


onMount not being called with esbuild

I experienced this problem whilst putting together https://github.com/Elderjs/elderjs/pull/217 but it always went away again if I did a clean install of @elderjs.
Now however, even if I clone the template as stated in the documentation with npx degit Elderjs/template elderjs-app, the onMount hook from svelte still isn't called when running the local server with esbuild. Resulting in the clock not ticking for example.
I have tried different node versions (12, 14, 17) and tried downgrading esbuild (even back to 0.12.4) but nothing seems to help.
Using rollup with yarn dev works fine...
Do you have any idea what the problem might be? Or where I could carry on digging?

Sorry for the delay on the PR. Been OOO a lot recently.
Make sure you’re in. A Svelte component and not a route or layout svelte file. Try that if you can.

No worries, but good to know you've seen it!
Well it is the default template that I am currently "testing" with. The clock component there is definitely a Svelte component and you use it to demonstrate component hydration...
... which was working for me perfectly 2 weeks ago, apart from my adventures into changing the default src directory for esbuild.
But now, with a fresh checkout and install, it doesn't. Only if you use rollup instead of esbuild.

Just asked a colleague to try as well and he confirmed, esbuild doesn't call the onMount hook.
I'm thinking it must be a dependency that has updated seeing as nothing in elderjs has changed. My first bet was esbuild itsself, but unless I did something wrong, downgrading it didn't solve the problem.

Super strange, fresh repo to test and can't recreate.
Any errors in the console?
Can you find the mount code?
It looks like so in the default for me:
   <script type="module">
            const requestIdleCallback = window.requestIdleCallback || ( cb => window.setTimeout(cb,1) );

const $$ejs = (par,eager)=>{
  const $ejs = function(_ejs){return _ejs};
  const prefix = '/_elderjs';
  const initComponent = (target, component) => {
    const propProm = ((typeof component.props === 'string') ? fetch(prefix+'/props/'+ component.props).then(p => p.json()).then(r => $ejs(r)) : new Promise((resolve) => resolve($ejs(component.props))));
    const compProm = import(prefix + '/svelte/components/' + component.component);

    Promise.all([compProm,propProm]).then(([comp,props])=>{
      new comp.default({
        target: target,
        props: props,
        hydrate: true
      });
    });
  };
  const IO = ('IntersectionObserver' in window) ? new IntersectionObserver((entries, observer) => {
    entries.forEach(entry => {
      if (entry.isIntersecting) {
        observer.unobserve(entry.target);
        const selected = par[entry.target.id];
        initComponent(entry.target,selected)
      }
    });
  }, { rootMargin: "200px",threshold: 0}) : undefined;
  Object.keys(par).forEach(k => {
    const el = document.getElementById(k);
    if (!eager && IO) {
        IO.observe(el);
    } else {
        initComponent(el,par[k]);
    }
  });
};

      requestIdleCallback(function(){
        $$ejs({'clockJZtQvlQQKb' : { 'component' : 'Clock.4YSKKBRU.js', 'props' : {}},})}, {timeout: 1000});</script>

Can you confirm that esbuild is outputting the code into your [distDir]/_elderjs/svelte/components/ folder for the clock?

OK, got it to happen. Weird. Appears to be some sort of weird race condition caused by the browser reload plugin.
Try this in your elder.config.js:
    '@elderjs/plugin-browser-reload': {
      // this reloads your browser when nodemon restarts your server.
      port: 8080,
      delay: 1000,
      reload: true, // if you are having issues with reloading not working, change to true.
    },
    ```

Next time it happens, try to manually reload the page and see if it is still happening. For me that is how I caught the issue.

I definitely tried manually reloading the page and hard refreshing. I'll check the other things now

Mount code is there:
    <script type="module">
            const requestIdleCallback = window.requestIdleCallback || ( cb => window.setTimeout(cb,1) );

const $$ejs = (par,eager)=>{
  const $ejs = function(_ejs){return _ejs};
  const prefix = '/_elderjs';
  const initComponent = (target, component) => {
    const propProm = ((typeof component.props === 'string') ? fetch(prefix+'/props/'+ component.props).then(p => p.json()).then(r => $ejs(r)) : new Promise((resolve) => resolve($ejs(component.props))));
    const compProm = import(prefix + '/svelte/components/' + component.component);

    Promise.all([compProm,propProm]).then(([comp,props])=>{
      new comp.default({
        target: target,
        props: props,
        hydrate: true
      });
    });
  };
  const IO = ('IntersectionObserver' in window) ? new IntersectionObserver((entries, observer) => {
    entries.forEach(entry => {
      if (entry.isIntersecting) {
        observer.unobserve(entry.target);
        const selected = par[entry.target.id];
        initComponent(entry.target,selected)
      }
    });
  }, { rootMargin: "200px",threshold: 0}) : undefined;
  Object.keys(par).forEach(k => {
    const el = document.getElementById(k);
    if (!eager && IO) {
        IO.observe(el);
    } else {
        initComponent(el,par[k]);
    }
  });
};

      requestIdleCallback(function(){
        $$ejs({'clockMmFnFfsVmX' : { 'component' : 'Clock.536SFERD.js', 'props' : {}},})}, {timeout: 1000});</script>

and setting a breakpoint in the instance function of the clock also gets triggered, but a breakpoint in onMount doesn't.
function instance($$self, $$props, $$invalidate) {
	let hours;
	let minutes;
	let seconds;
	let time = new Date();

	onMount(() => {
		const interval = setInterval(
			() => {
				$$invalidate(3, time = new Date());
			},
			1000
		);
...

Can you confirm that esbuild is outputting the code into your [distDir]/_elderjs/svelte/components/ folder for the clock?

yes, it is

I read somewhere about deduping svelte... just a shot in the dark, but could this have something to do with it?

Can you run npx envinfo --system --npmPackages svelte,rollup --binaries --browsers ? I'm wondering if the new version of Svelte could be causing it.

  System:
    OS: macOS 11.6
    CPU: (8) x64 Apple M1
    Memory: 48.16 MB / 16.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 12.22.3 - ~/.asdf/installs/nodejs/12.22.3/bin/node
    Yarn: 1.22.10 - ~/.asdf/shims/yarn
    npm: 6.14.13 - ~/.asdf/installs/nodejs/12.22.3/bin/npm
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 95.0.4638.69
    Firefox: 94.0.1
    Safari: 15.0
  npmPackages:
    rollup: ^2.41.1 => 2.59.0
    svelte: 3.43.2 => 3.43.2

yes!!! downgrading to 3.42.6 fixes it for me

going to 3.43.0 breaks it again. So something must've changed in that version that is causing this behaviour

https://github.com/Elderjs/elderjs/blob/bf528f1a20da10586467ce6c891d5bf3eafe3c79/src/esbuild/esbuildBundler.ts#L158
This should be "browser".


Cursor randomly disappear when using Gdx.input.setCursorImage on desktop/Windows

When using Gdx.input.setCursorImage the cursor can randomly disappear forever (the application needs to be closed, another call to Gdx.input.setCursorImage doesn't solve the problem)
It doesn't seem to happen at all under my Linux environment, just under Windows
Tested on:

Windows 64bits, jre 8 64 bits : ko
Windows 64bits, jre 7 64 bits : ko
Ubuntu 64bits, jre 7 64 bits : ok

To reproduce create a new project (tested with LibGdx 1.5.2), and change the main class content to:
Pixmap		pixmap	= null;

@Override
public void create()
{
	pixmap = new Pixmap(32, 32, Format.RGBA8888);
	pixmap.setColor(Color.GREEN);
	pixmap.drawCircle(15, 15, 10);
}

@Override
public void render()
{
	Gdx.gl.glClearColor(1, 0, 0, 1);
	Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);
	for (int i = 0; i < 1000; i++)
		Gdx.input.setCursorImage(pixmap, 16, 16);
}

Launch the application and just move the mouse around
Of course this code is a bit overkill but this will force the bug to appear quite quickly (usually under 10 seconds on my test environement)

Out of curiosity, when do you need to set that many time in every render ? I tried your code and I have the same bug but if I try with 10 as the loop boundary I don't have any problem.

This test is just for the bug to come quicker, of course it's not a real life example.
In my game the cursor changes to indicate weapon is reloading or ready to fire, basically it changes maybe 10 times per minute. The bug appear after about 1 hour of gameplay, and makes the game unplayble then.
As I don't want to wait one hour to reproduce this bug I just made it quicker in this test, but the idea is the same.

After digging a bit into the code it seems to me that the problem is that every time setCursorImage is called it creates a new lwjgl Cursor object.
And if I create lots of Cursor objects manually the cursor will disappear.
So it's either a bug in

lwjgl: it should be able to handle multiple Cursor creation, but it might be a system limitation
or in Libgdx implementation: there should be only one cursor per pixmap, maybe Libgdx shouldn't create a Cursor object every time (usiong a cache system or something).

So I'm now creating the Cursor objects myself and call lwjgl functions directly. Everything fine for me doing that.

Can you make a PR?
On Feb 13, 2015 9:23 PM, "haedri"<EMAIL_ADDRESS>wrote:

After digging a bit into the code it seems to me that the problem is that
every time setCursorImage is called it creates a new lwjgl Cursor object.
And if I create lots of Cursor objects manually the cursor will disappear.
So it's either a bug in

lwjgl: it should be able to handle multiple Cursor creation, but it
might be a system limitation
or in Libgdx implementation: there should be only one cursor per
pixmap, maybe Libgdx shouldn't create a Cursor object every time (usiong a
cache system or something).

So I'm now creating the Cursor objects myself and call lwjgl functions
directly. Everything fine for me doing that.
—
Reply to this email directly or view it on GitHub
https://github.com/libgdx/libgdx/issues/2839#issuecomment-74320551.

Assumed fixed by referenced PR.


Fix memoization for rails's server port value

I think multiple instance of AppBuilder are currently being initialized during suspenders APP_NAME process because bin/setup and development.rbhave different port values.
So I think we can use a class variable to memoize the value. Is it dirty?

I think multiple instance of AppBuilder are currently being initialized during suspenders APP_NAME

I add two questions: Why would be initialize more than one AppBuilder? Also surprising to me is the sample; why a sample, instead of the default or a given parameter?

Why would be initialize more than one AppBuilder?
I don't know why :(
why a sample, instead of the default or a given parameter?
It was introduced here: https://github.com/thoughtbot/suspenders/commit/4caa6123c83e726d6fcd23956ce9883ca36b1d42

In https://github.com/thoughtbot/suspenders/issues/602#issuecomment-130671649 we discussed the bug we are working around here. @jferris what do you think of simplifying suspenders, making it use default port 3000 always?

What do you think of simplifying suspenders, making it use default port 3000 always?

I have no objection to this, as I generally only run one project at a time. I'm not sure if most people run one project at once or several.
That change was introduced by @croaky, because I think he's in the habit of keeping several projects running simultaneously.
@croaky - any thoughts on this?

Yup, I run multiple projects at once for long periods of time.
I don't feel strongly either way. Do what you feel.

@delphaber do you want to remove the code around random ports from suspenders?

Ok, I'm gonna leave it 3000 as default

Done in https://github.com/thoughtbot/suspenders/pull/611
I'm closing this.