Adding new fields to IndexSegment and IndexValue

Adding reset key(StoreKey and Index Entry type) and lastModifiedTime (persist) to IndexSegment
See https://github.com/linkedin/ambry/issues/430 for further details

Changes and additions to IndexValue

OperationTime for every entry: This is required so that store statistics can reliably know when a delete or put has been issued to calculate the valid data size on any given time.
ServiceId and ContainerId: Every blob is expected to have a serviceId, containerID pair.
Expiration changed from ms to seconds precision. Or in other words, moved from 8 bytes to 4 bytes. See https://github.com/linkedin/ambry/issues/539 for further details
Introduced IndexValueBuilder to assist in constructing IndexValue as there were too many over loaded constructors and it was confusing too.

SLA: 30 to 40 mins
Reviewers: Gopal and Ming
Coverage:
IndexValue	  100% (2/ 2)	95.7% (22/ 23)	93% (80/ 86)
IndexSegment	  100% (1/ 1)	100% (25/ 25)	87.7% (322/ 367)
PersistentIndex 100% (7/ 7)	98% (49/ 50)	92.4% (532/ 576)
Testing:

Entire IndexValueTest is tested for both versions.
Entire IndexSegmentTest is tested for both versions.
Index roll over is tested when there is a change in IndexValue size.

@vgkholla have addressed all your comments
@xiahome : Can you review the patch when you get time.

@xiahome @vgkholla : have addressed all your comments.

Doesn't a delete op come with a time stamp from the router that gets
persisted into he delete record? If not can we make it so to be consistent.
On Feb 22, 2017, at 23:08, Sivabalan Narayanan<EMAIL_ADDRESS>wrote:
@nsivabalan commented on this pull request.
In ambry-store/src/main/java/com.github.ambry.store/IndexValue.java
https://github.com/linkedin/ambry/pull/541#discussion_r102483947:

import com.github.ambry.utils.Utils;
import java.nio.ByteBuffer;

/**

Represents the blob value stored in the index for a key.

Version_0

| Blob Size |   Offset  |  Flags  | Expiration Time | Orig msg  |

| (8 bytes) | (8 bytes) | (1 byte)|     in Ms       | offset    |

|           |           |         |     ( 8 bytes)  | (8 bytes) |

Version_1

@xiahome https://github.com/xiahome : Refers to the time in which the
operation was done. In other words, creation time for a put record and the
system time at which a delete was issued for a delete record.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/linkedin/ambry/pull/541#discussion_r102483947, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARk55ok1tkMGAyfL7K2WtjXxDdUM_rwZks5rfE9YgaJpZM4L9PrX
.

Delete records do not come with a timestamp today. It is not a "quick" change to make as it might need changes to the protocol and message format.

Haven't properly gone through this PR yet, but we have to ensure that anything that is added to the Index is also going to the Log. Log is the source of truth and during recovery the store will have to reconstruct Index entries from the entries in the Log, so if Operation Time, Container Id, etc. are not yet present in the Log, I would think these will have to go in first before they are included in the Index.
I also believe that while having to make a V3 is not ideal, it is not that big of a deal that we try to include all that we can think of and possibly want in the future to V2 when the imminent requirement from V2 is simply the Operation time. Note that by having default or null values for many of these fields - because we don't have them yet - we are going to effectively and permanently having implicit V2/V3 control flows.

@xiahome @nsivabalan Just a reminder to not merge this before #553 is merged.

@asinghai : Priyesh has answered your question. If you need more details, we can talk offline.
@pnarayanan :
This is the status for now:
All put records in index will assume defaults for operation time(-1), serviceId(0) and containerId(0).
Delete records are using time.milliseconds() as operationTime(in this patch). This will result in every replica having its own operationTime for the same blob. But again, making the changes what ashish suggested will be a separate PR in itself. So either we can leave it as is, or use -1 as default operationTime for delete records too.
Follow up patch1:
Add operation time to MessageInfo(creationTime incase of Put and operationTime in case of delete which the router sets) and hence all put records and delete in index will take up this value. At the same time, delete record in log will also have the operation time written to it.
Follow up patch2:
When serviceId and containerId design is complete and when we have a clear picture of how things go wrt serviceId and containerId, we can make changes to messageInfo to hold serviceId and containerId too. This patch will add serviceId, containerId info to log and index from the messageInfo.

@vgkholla : addressed all your comments. Do review it when you get time.

@pnarayanan
After giving a good thought, guess going with default value of -1 for delete op time makes sense for now. So, I have made changes in the patch itself to reflect that.

@vgkholla : addressed your recent comments

One minor comment. LGTM
Once you have made the final changes, please confirm formatting and build on OSx and Linux.

built in both OS and applied coding style.


dropdown: testing sweeto burrito

Bug
Expected Behavior
menu needs the drop
Actual Behavior
not dropping
Steps to Repro

drop
walk away

Related Packages

<EMAIL_ADDRESS><EMAIL_ADDRESS>
Environment

OS - MacOS 10.12
Browser version - Chrome v60

done

oops, nm, this was a test issue for slackbot.


Fix for #511

This is not a comprehensive fix but it should work for now.

Current coverage is 96.27%

Merging #512 into dev will decrease coverage by -0.04% as of 3cfb455

@@              dev    #512   diff @@
=====================================
  Files          23      23
  Stmts        2281    2282     +1
  Branches      206     207     +1
  Methods         0       0
=====================================
  Hit          2197    2197
- Partial        35      36     +1
  Missed         49      49

Review entire Coverage Diff as of 3cfb455

Powered by Codecov. Updated on successful CI builds.

@samkuehn OK. Thanks for this. Good work.


Set LAST_MESSAGE_ANCHOR to<PHONE_NUMBER>0000000 (sixteen zeroes).

Fixes: #3654.
The server correctly interprets this special number as indicating
that we want the latest messages (0 is used to request the oldest
messages). Later, we can conditionalize on the server version to
support the updated API merged in
https://github.com/zulip/zulip/pull/13747, that use the strings
'oldest' and 'newest' instead of 0 and<PHONE_NUMBER>0000000.

Thanks @chrisbobbe !
One point in the commit message: in the API, 0 doesn't mean the oldest unread messages, but simply the oldest. The legacy API for "oldest unread" is a separate boolean flag use_first_unread_anchor: true, which causes anchor to be ignored.
... Looking closer through our code for this, now that I have those facts about the API in my head from reading zulip/zulip#13747 recently, I see that within our code we do it the way the commit message describes, though. (And I guess we never make a request for simply the oldest messages.) It's confusing that we do something that's so almost, but not quite, like the server's API. I may see if there's a quick refactor I can do to clean that up.
Anyway, I'll make that adjustment in the commit message and then merge.

And pushed 7a3a84547 to try to clean up that confusion a bit:
commit 7a3a845477127fdb7a10b2dd682650c06f93ebaf
Author: Greg Price<EMAIL_ADDRESS>Date:   Thu Feb 6 20:45:28 2020 -0800

    anchor [nfc]: Correct the jsdoc on FIRST_UNREAD_ANCHOR.

    This jsdoc didn't match the name at all!  It turns out the name was
    right... and the jsdoc described what, confusingly, the very same
    value means in the actual Zulip API.

I also explored a refactor to actually stop using 0 in this confusing way. I have a draft that more or less does it, but more work is required before that's mergeable.


Create a diagram of the CKF bundle

Context
Create a diagram of the project and how all the components from the default CKF bundle work together so it can be added to the reference architecture public docs.
What needs to get done

Create a diagram of the charmed Kubeflow bundle: high-level diagram of the components, their integration points and dependencies.
Add the diagram to a reference architecture draft document

Definition of Done
There is a diagram of the CKF components (not published)

Closing as the diagram is done and added to the architecture documentation.


Implement: Package page operator area component

The Operator Area is part of the Package Page Header. This area should be used to display essential high level and distinctive information that should help a user make a basic filtering decision.
This information may be extracted from either the jsii.json or the metadata.json file.
Namely:

Package Name
Short Description
Published Date
Author
GitHub links
Dependencies
License

A user can click the Version drop-down menu to choose another version of the construct

We do not have this information easily available in the backend files right now. Decide if we skip this or add it to the backend somehow.

Merged with https://github.com/cdklabs/construct-hub-webapp/issues/43


docker badge showing invalid

issue

@ashish-patwal sure, go ahead

@ashish-patwal sure, go ahead

Thanks !!


[Public Catalog] handle many dashboards with many images better

Relates to #1541
On the external site when a quick start has many dashboard images of different name types, we currently show :one:  them like this.  Whereas when we have many dashboards images of the same name type we show :two: them like this.
So  quick start can have N dashboards with N Images.
Acceptance Criteria

[x] stack carousels per dashboard name type
[x] add a line between each carousel
[x] Be careful of is making sure the height of none of those carousels gets too tall

Bug

[x] ensure we fix the bug where a carousel control dot is appearing next to a screen shot.)

This work has been completed 🎉


release-please failed: Cannot convert undefined or null to object (package-lock.json with lockfileVersion: 1)

Thanks for stopping by to let us know something could be better!
Please provide the following details.
Environment details

OS: Ubuntu 22.04 LTS (GitHub Public Runner)
Node.js version: 18.20
npm version: 10
release-please version: 4

Steps to reproduce

Create a package-lock.json using lockFileVersion: 1 (see this example)
Run release-please.

Expected
Error message:
Your `package-lock.json` uses lockFileVersion X, which is not supported by release-please.

Actual
Cannot convert undefined or null to object

See this action run.
Context
https://github.com/googleapis/release-please/blob/08331fa555ed482f675fb3e9dd5b0a5296e200a3/src/updaters/node/package-lock-json.ts#L46

+1, facing the same issue with package-lock.json having lockfileVedrsion == 1


link codebook to ANES data

The .dta files read into pandas via read_stata(). The codebook contains descriptions of variables names, question text, etc. Once this is parsed, it will need to be linked with the response data.

They are linked via the VCF code keys VCFxxxx.


Port utils::gentabs to Rust

This commit ports utils::gentabs to rust, as outlined in #334. The original tests pass.
As this is my first attemt at production rust, any comments and criticism is welcome :)
I'm not really convinced about all that casting in the rust implementation, even though it should not really be any practical problem (both isize and usize should be able to store all effective tab-values that should occur. Even theory-wisse as stringwidth is bounded by the addressable memory, this implementation should work, but it still seems strange to have all that casting going on).

Coverage increased (+0.5%) to 40.691% when pulling 6741b5174e3dd92c3fc4e7c1a67672d714b5d5a1 on noctux:gentabs into c63da29f70de21f045a1e40d45bcbf332ef71a4c on newsboat:master.

Yeah, the casting isn't too pretty. But the fact that it's explicit warms my heart a bit :) I'd simply rewrite this like that:
let tabcount = strwidth(string) / 8;
if (tabcount >= 4) {
    1
} else {
    4 - tabcount
}

I'm ready to merge this as-is, so if you don't want to tweak it any further, let me know. Thanks for the work!

Hey, I like your version :) I think I did cling too much to the C++ variant. Nice how one can get rid of all that casting :P I've switched to that variant now. Thanks for the review!


Support wallet v2  apis / auth so we can login as market-sim parties into console

Support wallet v2  apis / auth so we can login as market-sim parties into console.
Console now uses the wallet v2 api(s) which means we cannot use the current setup to login to the console as one of the agents we created :-( . Let's fix that; it's very useful ;-).

Blocked by https://github.com/vegaprotocol/vega/issues/6819


More robust validation of createTx params

Coverage remained the same at 80.987% when pulling 82bd5888340f7a8a7dbbc1b60b98286008c0e106 on isocolsky:fix/args-check into 4dc010cf0c7c4de6c6b9d8652a1394ec27aab49d on bitpay:master.


[WIP] Separate loading drives and loading connectivity

As titled.

Codecov Report

Merging #566 (7f3760a) into master (908c818) will decrease coverage by 0.04%.
The diff coverage is 93.75%.

:exclamation: Current head 7f3760a differs from pull request most recent head 035904f. Consider uploading reports for the commit 035904f to get more accurate results

@@            Coverage Diff             @@
##           master     #566      +/-   ##
==========================================
- Coverage   91.43%   91.38%   -0.05%
==========================================
  Files          22       21       -1
  Lines        4157     4145      -12
==========================================
- Hits         3801     3788      -13
- Misses        356      357       +1

Impacted Files
Coverage Δ

hnn_core/gui/gui.py
95.26% <93.75%> (+0.71%)
:arrow_up:

hnn_core/parallel_backends.py
81.38% <0.00%> (-0.84%)
:arrow_down:

hnn_core/cell_response.py
84.00% <0.00%> (-0.54%)
:arrow_down:

hnn_core/extracellular.py
86.69% <0.00%> (-0.26%)
:arrow_down:

hnn_core/cell.py
97.10% <0.00%> (-0.11%)
:arrow_down:

hnn_core/gui/_viz_manager.py
87.36% <0.00%> (-0.10%)
:arrow_down:

hnn_core/__init__.py

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@chenghuzi let us know when this is ready!

@chenghuzi let us know when this is ready!

Sorry for the delayed response! It had already been MRG last time but I forgot to update the title for a while. I just refined the style a little bit and marked it as MRG now.

Code looks good @chenghuzi!
Going to take a moment to just click around the GUI but will merge soon.


x/tools/internal/lsp: can't find import

What version of Go are you using (go version)?

┏━ oneofone@Ava ❰~❱
┗━● go version
go version devel +dbd323bb88 2018-12-14 18:15:56 +0000 linux/amd64
┏━ oneofone@Ava ❰~❱
┗━● git -C $GOPATH/src/golang.org/x/tools describe --always
57eff0d8

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
┏━ oneofone@Ava ❰~❱
┗━● go env
GOARCH="amd64"
GOBIN="/home/oneofone/code/go/bin"
GOCACHE="/tmp/.gocache"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/oneofone/code/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/src/go"
GOTMPDIR=""
GOTOOLDIR="/usr/src/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build911290659=/tmp/go-build -gno-record-gcc-switches"

What did you do?
Working on a rather large workspace in vscode + vscode-go 0.8.0 + latest golsp as of 10 mins ago, I keep getting can't find import on "one" specific internal package.
However, it works fine except for printing that error (and marking the import as an error in the editor), it auto completes the package and types in it just fine.
Deleting the $GOCACHE folder doesn't change anything.
Editor config
	"go.alternateTools": {
		"go-langserver": "golsp"
	},
	"go.useLanguageServer": true,
	"go.languageServerExperimentalFeatures": {
		"format": true,
		"autoComplete": true,
		"rename": true,
		"goToDefinition": true,
		"hover": true,
		"signatureHelp": true,
		"goToTypeDefinition": true,
		"goToImplementation": true,
		"documentSymbols": true,
		"workspaceSymbols": true,
		"findReferences": true
	},

What did you expect to see?
No errors.
What did you see instead?
[Trace - 11:57:57 PM] Sending request 'textDocument/codeAction - (65)'.
Params: {"textDocument":{"uri":"file:///home/oneofone/code/go/src/github.com/PRIVATE/service/service.go"},"range":{"start":{"line":21,"character":2},"end":{"line":21,"character":2}},"context":{"diagnostics":[{"range":{"start":{"line":21,"character":2},"end":{"line":21,"character":2}},"message":"could not import github.com/PRIVATE/internal/hose (can't find import: \"github.com/PRIVATE/internal/hose\")","severity":1,"source":"go"}]}}

@ramya-rao-a  @stamblerre

@OneOfOne: this sounds like a diagnostic being generated by the VSCode-Go extension - if you notice, the "source" is set to "go", whereas golsp always sets the source to "LSP" (https://github.com/golang/tools/blob/ae5b88167698f5f718baac67cf6088827de6569f/internal/lsp/diagnostics.go#L53). You can look at the diagnostics that golsp is sending by looking for the "publishDiagnostics" messages in the logs.


Bump yard gem to 0.9.20

There are two security vulnerabilities in the currently installed version of yard [1] and [2].
Therefore bumping the gem to the minimum version that includes patches for these issues.
1: https://github.com/alphagov/govuk_schemas/security/dependabot/1
2: https://github.com/alphagov/govuk_schemas/security/dependabot/2
Trello card

I also wonder if we just want to remove this dependency, it's only reference is that you can run a yard server in the docs and I very much doubt anyone has done that for many years.

That seems reasonable, since I'm not aware of anyone actually running this server locally.
I'll close this PR and raise another to remove yard (and the associated docs).

@kevindew Alternative PR: https://github.com/alphagov/govuk_schemas/pull/84


Provider instance is created even when MetaMask is not installed

I am using this package to work around a Firefox issue where the eth provider cannot be injected: https://github.com/MetaMask/metamask-extension/issues/3133
On Chromium with no metamask extension, this package still creates a full provider object, though some of the properties are undefined.
I am just wondering what is the correct way to check if the provider is good to use.  I am planning for now to check the value of chainId, but I don't know if this is actually a bug.
Here is the console.log output for the provider:
Proxy {_events: {…}, _eventsCount: 1, _maxListeners: 100, isMetaMask: true, _state: {…}, …}
[[Handler]]: Object
deleteProperty: () => true
__proto__: Object
[[Target]]: MetamaskInpageProvider
autoRefreshOnNetworkChange: false
chainId: undefined
enable: ƒ ()
isMetaMask: true
networkVersion: undefined
request: ƒ ()
selectedAddress: null
send: ƒ ()
sendAsync: ƒ ()
_events: {connect: ƒ}
_eventsCount: 1
_handleAccountsChanged: ƒ ()
_handleDisconnect: ƒ ()
_maxListeners: 100
_metamask: Proxy {isUnlocked: ƒ, requestBatch: ƒ, isEnabled: ƒ, isApproved: ƒ}
_publicConfigStore: ObservableStore {_events: {…}, _eventsCount: 1, _maxListeners: undefined, _state: {…}}
_rpcEngine: RpcEngine {_events: {…}, _eventsCount: 0, _maxListeners: undefined, _middleware: Array(3)}
_rpcRequest: ƒ ()
_sendSync: ƒ ()
_state: {sentWarnings: {…}, isConnected: true, accounts: undefined, isUnlocked: undefined}
_warnOfDeprecation: ƒ ()
_web3Ref: undefined
__proto__: SafeEventEmitter
[[IsRevoked]]: false

It's possible that I did have metamask installed at one point and have deleted since then.  I am looking around chrome://about to see if I can find any local data that might be left behind, but I don't know exactly where it would be.

There's an event pair, connect/disconnect, and a related method, isConnected, that are supposed to be used for this purpose, but unfortunately they're not correctly implemented yet. I'll keep this issue open until they are.
Meanwhile, checking if the chainId is undefined is a good solution. The chainId is only populated if the provider successfully connects to MetaMask, so its absence implies the absence of MetaMask.


Fix multiple buffer overflows and a buffer underflow.

Hello,
Upon investigation there are multiple places where buffer-overflows could occur aswell as one case of a potential underflow in strip().
The solution i propose is: instead of using sprintf directly without checking the if the buffer overflows or using nsprintf and adding a bunch of code for checking the return value, add csprintf (checked sprintf) which throws an exception in case the buffer is too small.
This change requires some changes in function signatures to propogate the size of the buffer used (notably in the Serial module).
The echo function is a special case which should never fail (since the device running lizard could become "mute"). Therefore the output will get truncated if the buffer overflows.
Ideally, the echo function should not truncate either. This would require the use of vprintf.
The drawback would be the potential of blocking the main thread in case the output buffer of the uart driver is full (the potential is already there if the uart driver buffer has not been cleared until the next echo()).
Kind regards

I want to add, for clarity, that the return of sprintf, snprintf and vsnprintf says how much would have been written to the buffer (in case of sprintf the actual amount written).
It is therefore necessary to check the return values of snprintf and vsnprintf in more detail.
The echo() overflow was caused by this behavior, as the pos value added by vsnprintf to pos could exceed the length of the buffer (whilst not actually writing past the buffer). The issue arises when adding the newline char in the following sprintf call, where the offset then could exceed the buffer size.

There is one more potential issue (which might warrant its own issue/pr).
When the function Serial::read_line is called and there are more than 2 lines in the uart buffer, the pattern positions returned by uart_pattern_get_pos and uart_pattern_pop_pos for the second pattern position can be incorrect, when the buffer is read in 2 different read_line calls.
As far as I could see, the pattern positions never get updated when calling uart_read_bytes.
Therefore, all currently detected patterns must be handled even if only one line is supposed to be read.

After another thought about the min value in the echo() function:
The max value for num_chars is set to sizeof(buffer) - 1 (which is 1023 currently). That would mean that the string termination gets put at 1024 (outside of buffer).
vsnprintf can write at most n-1 characters into the buffer. n is sizeof(buffer)-1, thus 1023, so max 1022 chars are written with the termination at 1023 (correctly leaving space for the newline).
If the vsnprintf writes 1022, but returns 1023 or above (since it returns the amount of chars it would have written), the newline gets written at index 1023 and the termination at 1024 (buffer overflow).
It should be std::min(num_chars, static_cast(sizeof(buffer) - 2)) instead.

If the length of the received line in the uart buffer exceeds the buffer size given to Serial::read_line, we cannot read the line.
There are 2 options to handle this:

Discard the line.
Wait for another call to Serial::read_line with sufficiently sized buffer.

I picked the first option since we don't want the serial to clog up, but neither way is ideal. The if(serial.available() < pos) {} block is for flushing the serial buffer and pattern positions if the pattern position is invalid. (This somewhat more related to the my previous comment about the pattern positions not getting updated when reading.)
Although throwing an exception might be a better idea.

I am unsure if the function can actually determine the size of the buffer themselves. As far as I know sizeof is calculated at runtime and using sizeof in the functions would result in getting the size of a char pointer. I would like to agree, but I don't think its possible.


"NOT FOUND "completion behavior in menu when there is only matching 1 suggestion

holo-cli does not have an expected behavior while completing word when there is only one matching solution.
Example with interfaces -> interface -> <interface-name> ->ip<?>:
Here completion 2 alternatives:

The last completion should complete to ipv4:

However, this is a general problem, when only there is one suggestion left in the completion list and that the cursor postion is right fater the matching word, the menu shows NO RECORDS FOUND. (It should be NO RECORD FOUND BTW)

Closed by #7


Fine grained rate-limits

acmed currently allows setting rate limits on endpoints, that specify how many
requests can be made per interval. It does not take the kind of request that is
made to the endpoint into consideration, which means that the rate limits
actually set by CAs (see https://letsencrypt.org/docs/rate-limits/) can not
really be represented by this.
My suggestion is to extend the current mechanism by giving rate-limits an
optional scope. This scope can be the path in the HTTP request, a path prefix,
an ACME resource (as listed in the directory response), etc. A request is then
only blocked, if there is a currently blocking rate-limit on that endpoint, that
matches this request. The specifics on how to model that and how to expose the
configuration for this aren't trivial, but I think if at least path (prefixes)
and ACME resources can be used as a scope, a few important rate limits can now
be set properly, that couldn't be set before, like the one for the newOrder
resource (300/3h), a shared one for the ACME resources newNonce, newAccount,
newOrder and revokeCert (20/1s), and the one for /directory
and anything under /acme (40/1s).

I've started looking at this from an implementation side for a bit here. Combining multiple rate-limits is a bit tricky apparently, looking at different implementations (like https://lib.rs/crates/htb and https://lib.rs/crates/governor) they either don't implement them (see https://github.com/antifuchs/governor/issues/156), or not in a way that's flexible enough for us (htb says that buckets need fixed parents, which makes configuring this a lot less versatile than what I've described above).
I'll take a deeper look at some point, but I won't continue here tonight.

Sleeping over it: Maybe it makes more sense to go for an implementation that somewhat works first before going for an implementation that works perfectly. For now, that will just mean that I'll be waiting for a positive outcome on each of the limits that match a request in sequence, ordered by the refill rate (from slow to fast), which is a good enough approximation IMO.


Chore: bump version of some dev dependencies

… and temporarily disable coveralls in main push action
Plz review @terrestris/devs

Superseeded by #500.


Feature/#1098 events

Implementation of AmpersandTarski/Ampersand#1098

@hanjoosten could you review this PR? Interested in what you think and if you have any suggestions. Or just approve if it is ok with you to add this.


Color sensor on RVR not working

I have tried to get the color sensor working on the Sphero RVR. I can see that the enable_color_detection code (56) is being sent to the RVR but the dictionary data streaming back doesn't contain any "color_detection" data.

This is a little harder for me to repro currently as I don't have an RVR. I can take a look this weekend.


I (12271) QR: quirc_resize err

Hi:
I run the QR task, it prints the error "QR: quirc_resize err", wired :(  I didn't meet it before.
Can you tell me why ? and how to fix it , Thanks :D

I encountered the same problem and found that the quirc_resize function in quirc.c reported an error. The application for the memory space function calloc failed, returning NULL, I don't know how to solve it.

You should use ESP32 module with psram

Boa tarde tem como usa sua biblioteca no ide arduino so que com esp32cam


Windows Defender Flags the .exe file as a Trojan

Windows Defender is flagging the .exe as a Win32/Zpevdo.B Trojan.
When running: https://i.imgur.com/TWoeupu.png
When scanned: https://i.imgur.com/NkDngon.png

I also had this problem and Windows defender deleted the exe file.  I tried to download the zip again and it also flagged that and deleted the download.

Same issue.

Go to Start > Settings > Update & Security > Windows Security > Virus & threat protection. Under Virus & threat protection settings, select Manage settings, and then under Exclusions, select Add or remove exclusions. Select Add an exclusion, and then select FallGuystStats.exe

this only happened starting today. I think the github download may have been hacked and had a trojan horse inserted.

Same Happened to me. What are the chances it's a real virus vs a false detection?

I'm leaning towards a false detection.  I'm pretty sure I updated and used the tracker around 1am without problem.  I also ran a scan with Avast.  I guess we can be sure if someone downloads and builds the source and compares the files.

Likely false detection. Last commit was 3 days ago and nothing there in the commit that is even close to being scary.

One of the possible solutions is adding the author and description in the assemblyinfo.cs

Yeah, I cloned the code and added in the AssumblyDescription and AssemblyCompany to some values and it stopped triggering it for me.

There is no virus in the program, would just be a false positive. I added more info to the AssemblyInfo as Eldorian suggested. So hopefully that helps.

@ShootMe the change allowed me to download it without triggering windows defender like it was earlier, so hopefully that helped.

Looks okay now.  I think you can close this issue after those changes.

I downloaded the latest update, and it ran without triggering Windows Defender.


Add optional path parameter for directory files list

Fixes #83
This adds an optional path parameter to :FzfPreviewDirectoryFiles command (and fzf-preview internal functions) that will be added at the end of g:fzf_preview_filelist_command before listing files.
This way we can for example open files preview of the directory of the current file opened like so
nmap <silent> 0 :FzfPreviewDirectoryFiles <C-R>=expand('%:h')<CR><CR>

Thanks.
Please add to README and doc.

And finally, please rebase from origin/master.

LGTM


Add the implementation of IMemoryManager to allow users to pool char[] and long[] arrays

SUMMARY
Inject the IMemoryManager instance whenever resources are created, and add close() methods to free allocated resources.
The default implementation changes nothing with how the library works, but users can implement their own manager which pools chars and longs.
Most changes are simple by just adding the memoryManager.
Discussion Thread: https://groups.google.com/g/roaring-bitmaps/c/V4-lRjraMJ0
Automated Checks

[x] I have run ./gradlew test and made sure that my PR does not break any unit test.
[x] I have run ./gradlew checkstyleMain or the equivalent and corrected the formatting warnings reported.

I read the email exchange. Have you considered memory mapping your bitmaps instead?

It would be important to show that this change comes at no cost for people who do not make use of this feature. So it should not increase memory usage or otherwise slow the code down.
It will surely make the code harder to maintain and more error prone. To compensate for this downside, we would need to be able to show better performance in at least some instances.
If these conditions are met, then we can possibly merge the PR if there is wide agreement in the community.
However, if these conditions are not met, you can still use your approach but as part of a fork of the project. We do not encourage forks… but if the changes only benefit you, then it is the best option for everyone.

It is really important for us to avoid regressions both in terms of performance and also in correctedness.

I read the email exchange. Have you considered memory mapping your bitmaps instead?

It would be important to show that this change comes at no cost for people who do not make use of this feature. So it should not increase memory usage or otherwise slow the code down.
It will surely make the code harder to maintain and more error prone. To compensate for this downside, we would need to be able to show better performance in at least some instances.
If these conditions are met, then we can possibly merge the PR if there is wide agreement in the community.
However, if these conditions are not met, you can still use your approach but as part of a fork of the project. We do not encourage forks… but if the changes only benefit you, then it is the best option for everyone.

How about I try adding

It would be important to show that this change comes at no cost for people who do not make use of this feature. So it should not increase memory usage or otherwise slow the code down.
It will surely make the code harder to maintain and more error prone. To compensate for this downside, we would need to be able to show better performance in at least some instances.
If these conditions are met, then we can possibly merge the PR if there is wide agreement in the community.
However, if these conditions are not met, you can still use your approach but as part of a fork of the project. We do not encourage forks… but if the changes only benefit you, then it is the best option for everyone.

I would need your help with that.
How do you think we should move forward, in terms of performance, theoretically it should not cause a regression, and the same applies on correctness.
The only footprint this adds is the memory added for references.
How about we create a CustomMemoryManager that pools the objects by default, and let the user decide if he wants to use it.
This should reduce significantly the temporary objects created if the user opts to use the new manager.
I can try it but I need a way to measure the effect.
How do you usually do this?

My concern with this implementation is the footprint overhead, which is up to 0.5MB for 32 bit bitmaps. This imposes bloat on every user, whether DIY memory management seems like a good idea to them or not. The cause of the bloat is the dependency injection style implementation.
What could work is an SPI style memory manager, from which all arrays are obtained. If the memory manager is stored in a static final field (with registration possible before first use) then the calls will all get inlined anyway and there will be no cost at all. Container, RoaringArray, and RoaringBitmap become AutoCloseable, and arrays are returned to the memory manager on calling close, or at resizes.
This has shortcomings - if two libraries in the same process want to use RoaringBitmap with different memory managers, they will not be able to: one of them will register their memory manager via the SPI first and the other will lose. This is unlikely though because the vast majority of users will not want to custom memory management… JVMs already have memory management in the form of TLABs and garbage collection, which in recent times will outperform DIY attempts. But it would be possible for a user to override the behaviour at no cost to other users this way, if they want to.
Features not imposing costs on users who don’t use the feature has always been a design principle in this library as things have been added, and I think it’s an important one.

Another very important step would be to demonstrate that you can outdo the Java memory manager in a realistic setting. As I made clear, this is not at all obvious. There are many sophisticated knobs to the memory managers, with respect to how you want them to work. You have to show that you can beat Java at its own game, with the best settings.
It is not at all obvious to me how you can do better by pooling char[] and long[] arrays in a realistic setting. The JVM has far more options and visibility than you do as a Java programmer. Beating it is hard. Memory allocation must also be thread safe in general, so this requires locking in many realistic applications (whereas the JVM can almost surely rely on lightweight concurrency).

This should reduce significantly the temporary objects created if the user opts to use the new manager.

It does not follow that you have better performance. You have to show that your application runs faster. You can, and probably should, write a benchmark using JMH.
Java is very well tuned to dealing with lots of temporary objects. If you reduce temporary objects while incurring no other cost, then you win, surely, but if you work hard to reduce temporary objects, you may very well end with a negative because you are competing against what the JVM does best.
Please be mindful that you should be able to win by a sizeable margin to justify the added complexity. E.g., you should be able to show that a reasonable realistic task can be achieved, say, at least  20% faster (with hard number).

I also feel the added complexity might not surpass the yet-to-confirm benefits. However, the spirit/initial motivation looks positive, and the PR generous. Maybe we could consider dropping some final keywords (e.g. to enable extending ArrayContainer), and wrap some code in dedicated methods (e.g. char[] allocations). It would enable such (and other) feature(s) without impacting core implementations.
The perf/memory impact of derivation might impact only these custom implementations (or not, e.g. I'm not telling wrapping char[] allocation in a non-final non-private method is free).
One would then offer such specific features through additional classes. We may accept them, or accumulate in some extras jar (as discussed previously), or prefer parking into an additional specific module, or in the examples, or suggest creating a specific library (on top of RoaringBitmap, instead of forking).
Indeed, if @alimans3  find any concrete benefit with his approach, we could ensure it can be applied without requiring him to fork RoaringBitmap core classes. hence, this PR may also turn into the minimal changes to make his approach (and others) applicable, without requiring duplicating full-classes/full-methods.

I just spent a little bit of time playing with an alternative, and wrote a reference counting test to make sure everything allocated is freed, and making Container closable will be extremely buggy. Containers are created in an ad hoc fashion, transitioned between in an unpredictable manner so being able to close every created container so it can release its memory is going to be fraught... There is finalize but let's not go there...

Additional classes would be fine. I also do not discourage this PR: see my comments on the mailing list.

It's also not something you can hope to do reliably without the kind of instrumentation points the GC has access to... Consider the case here where a bit is added to an ArrayContainer:
    if (newcardinality > DEFAULT_MAX_SIZE) {
      BitmapContainer a = this.toBitmapContainer();
      return a.iadd(begin, end);
    }

There are a few ways you can approach this. You can call this.close() before returning, but that would be wrong if this is stored in an array somewhere after creating the new bitmap container. What you actually need to do is intercept every single reference assignment to determine if the assigned reference is now dereferenced so it can be closed. This is what GC engineers call a store barrier, and this way lies madness.

I also do not discourage this PR: see my comments on the mailing list.

I have discouraged this PR not because I don't appreciate the effort that has gone in to it, I do, I just can't see it working and my concerns go way deeper than the footprint overhead now.

If anyone is interested, my code is here https://github.com/RoaringBitmap/RoaringBitmap/tree/memory-manager-prototype

@richardstartin I think @blacelle refers to specialized classes. I agree that re-engineering everything seems unwise.

@richardstartin I think @blacelle refers to specialized classes. I agree that re-engineering everything seems unwise.

@lemire you still can't guarantee to close everything without intercepting reference assignment (store barriers), custom classes or not.

I may have overstated my case a little bit, there are relatively few container assignments in RoaringArray and it's possible to route all of them through setContainerAtIndex - see #541

@richardstartin I think @blacelle refers to specialized classes.

Yep.

I may have overstated my case a little bit, there are relatively few container assignments in RoaringArray and it's possible to route all of them through setContainerAtIndex - see #541

I took a look at your PR, the approach you did is definitely better than what I started with.
The overhead is removed and the Allocator static methods are clearer and less infiltrating in the code.
Regarding the case you mentioned:
public Container add(int begin, int end) {
...
   if (newcardinality > DEFAULT_MAX_SIZE) {
     BitmapContainer a = this.toBitmapContainer();
     return a.iadd(begin, end);
   }
...

In this case the caller should be responsible of closing the old container if the new one replaces it.

Regarding the case you mentioned:
...
In this case the caller should be responsible of closing the old container if the new one replaces it.

The thing is that's easier said than done, it's really hard to guarantee. There's an awful lot of code where containers do not have well defined scopes. Take RunContainer.or and find all the places a container needs to be closed:
  @Override
  public Container or(BitmapContainer x) {
    if (isFull()) {
      return full();
    }
    // could be implemented as return toTemporaryBitmap().ior(x);
    BitmapContainer answer = x.clone();
    for (int rlepos = 0; rlepos < this.nbrruns; ++rlepos) {
      int start = (this.getValue(rlepos));
      int end = start + (this.getLength(rlepos)) + 1;
      int prevOnesInRange = answer.cardinalityInRange(start, end);
      Util.setBitmapRange(answer.bitmap, start, end);
      answer.updateCardinality(prevOnesInRange, end - start);
    }
    if (answer.isFull()) {
      // need to close answer here
      return full();
    }
    return answer;
  }

  @Override
  public Container or(RunContainer x) {
    if (isFull()) {
      return full();
    }
    if (x.isFull()) {
      return full(); // cheap case that can save a lot of computation
    }
    // we really ought to optimize the rest of the code for the frequent case where there is a
    // single run
    RunContainer answer = new RunContainer(allocateChars(2 * (this.nbrruns + x.nbrruns)), 0);
    int rlepos = 0;
    int xrlepos = 0;

    while ((xrlepos < x.nbrruns) && (rlepos < this.nbrruns)) {
      if (getValue(rlepos) - x.getValue(xrlepos) <= 0) {
        answer.smartAppend(getValue(rlepos), getLength(rlepos));
        rlepos++;
      } else {
        answer.smartAppend(x.getValue(xrlepos), x.getLength(xrlepos));
        xrlepos++;
      }
    }
    while (xrlepos < x.nbrruns) {
      answer.smartAppend(x.getValue(xrlepos), x.getLength(xrlepos));
      xrlepos++;
    }
    while (rlepos < this.nbrruns) {
      answer.smartAppend(getValue(rlepos), getLength(rlepos));
      rlepos++;
    }
    if (answer.isFull()) {
      // need to close answer here
      return full();
    }
    // may need to close answer here, because it may not reach the caller.
    return answer.toBitmapIfNeeded();
  }

I think if you add a reference counting test on this branch, you will realise how hard it is to retrofit this behaviour. The same goes for subclassing - there's a lot of places the subclass needs to be preserved through combining operations, some of which escape, some don't. The only surefire way to do this is to have the same store barriers the GC has.

Regarding the case you mentioned:
...
In this case the caller should be responsible of closing the old container if the new one replaces it.

The thing is that's easier said than done, it's really hard to guarantee. There's an awful lot of code where containers do not have well defined scopes. Take RunContainer.or and find all the places a container needs to be closed:
  @Override
  public Container or(BitmapContainer x) {
    if (isFull()) {
      return full();
    }
    // could be implemented as return toTemporaryBitmap().ior(x);
    BitmapContainer answer = x.clone();
    for (int rlepos = 0; rlepos < this.nbrruns; ++rlepos) {
      int start = (this.getValue(rlepos));
      int end = start + (this.getLength(rlepos)) + 1;
      int prevOnesInRange = answer.cardinalityInRange(start, end);
      Util.setBitmapRange(answer.bitmap, start, end);
      answer.updateCardinality(prevOnesInRange, end - start);
    }
    if (answer.isFull()) {
      // need to close answer here
      return full();
    }
    return answer;
  }

  @Override
  public Container or(RunContainer x) {
    if (isFull()) {
      return full();
    }
    if (x.isFull()) {
      return full(); // cheap case that can save a lot of computation
    }
    // we really ought to optimize the rest of the code for the frequent case where there is a
    // single run
    RunContainer answer = new RunContainer(allocateChars(2 * (this.nbrruns + x.nbrruns)), 0);
    int rlepos = 0;
    int xrlepos = 0;

    while ((xrlepos < x.nbrruns) && (rlepos < this.nbrruns)) {
      if (getValue(rlepos) - x.getValue(xrlepos) <= 0) {
        answer.smartAppend(getValue(rlepos), getLength(rlepos));
        rlepos++;
      } else {
        answer.smartAppend(x.getValue(xrlepos), x.getLength(xrlepos));
        xrlepos++;
      }
    }
    while (xrlepos < x.nbrruns) {
      answer.smartAppend(x.getValue(xrlepos), x.getLength(xrlepos));
      xrlepos++;
    }
    while (rlepos < this.nbrruns) {
      answer.smartAppend(getValue(rlepos), getLength(rlepos));
      rlepos++;
    }
    if (answer.isFull()) {
      // need to close answer here
      return full();
    }
    // may need to close answer here, because it may not reach the caller.
    return answer.toBitmapIfNeeded();
  }

I think if you add a reference counting test on this branch, you will realise how hard it is to retrofit this behaviour. The same goes for subclassing - there's a lot of places the subclass needs to be preserved through combining operations, some of which escape, some don't. The only surefire way to do this is to have the same store barriers the GC has.

You’re right, I think its hard to count all these possible cases.
However, failure to close (free) containers would result in it being collected by the GC (unless referenced by custom allocator which is counterintuitive), so the worst case would be having the same footprint of the current implementation.
And with time, the code should evolve to cover all the missing cases when the Allocator approach is adopted.
What do you think?

However, failure to close (free) containers would result in it being collected by the GC (unless referenced by custom allocator which is counterintuitive), so the worst case would be having the same footprint of the current implementation.

I don’t think we could merge this PR without a custom allocator that shows the utility of this strategy. And said allocator would have to be correct (no leak).

However, failure to close (free) containers would result in it being collected by the GC (unless referenced by custom allocator which is counterintuitive), so the worst case would be having the same footprint of the current implementation.

I don’t think we could merge this PR without a custom allocator that shows the utility of this strategy. And said allocator would have to be correct (no leak).

I will work on a benchmark that uses a custom allocator that pools objects to show the temporary allocations gain, once @richardstartin finalizes the other PR which is the dominant PR now.

Let me be clearer… We need a fully worked out, tested and benchmarked solution. We will not merge into our main branch something that it is not fully motivated.
Our core code is mostly bug free and is performant. People rely on it. We will not take chances.
I am not discouraging this work but I want us to understand that experiments must be validated before they get merged and publish. Engineers are relying on us for their production code.
If the code becomes more complicated then we need a tangible benefit. So we must see the code run faster!

@alimans3 I'm less concerned by leaks than use after free scenarios, but if you can't avoid guarantee leaks don't happen it's unlikely you can guarantee use after free doesn't. It's very easy to underestimate the complexity of doing this sort of thing properly, and the consequences of getting it wrong tend to be fairly severe.

@richardstartin @lemire @blacelle
I added a very simple, non-threadsafe, implementation of the allocator just to show the potential of this development.
I don't think we should provide the custom allocator implementation, this is just to be able to demo.
In my project code, We use a similar pool design, however We made it thread safe in a CPU efficient way.
I added a benchmark with some operations on random bitmaps and here are the results:
before.log
after.log
I ran the benchmarks with -prof gc to be able to get GC data, and you can find the obvious reduction in GC pause count, GC allocation rate, etc.
Let me know what you think.

@alimans3 that's more than a 10% regression on intersection speed. I don't think it's a good tradeoff given that JVMs are designed to allocate quickly and GCs are getting better and better and recycling allocated memory with low pause times.

@alimans3 that's more than a 10% regression on intersection speed. I don't think it's a good tradeoff given that JVMs are designed to allocate quickly and GCs are getting better and better and recycling allocated memory with low pause times.

This regression is a result of the implementation of the pool which i think is far from perfect.
I think it would be good if we give the user the ability to do this tradeoff if his application is memory-critical.
Even if the GC is becoming better, keeping on creating temporary objects, with large sizes in heavy applications, would reduce its performance.
in my application for example, long GC pauses and even application halts were encountered and this massively reduces it.

I understand the custom IMemorymanager leads to a 10% perf degradation (91 -> 100), but to better-looking GC statistics (.count, .time, @alimans3 I feel you should also focus on gc.time.max statistics). Hence, this implementation would also a trade-off one might be interested in, with limited perf-impact.
One of the big question remains: how does the implementation with default IMemoryManager behave compared to current implementation: @alimans3 , can you provide figure from your JMH run with the default/dummy allocator, and with master code?

I understand the custom IMemorymanager leads to a 10% perf degradation (91 -> 100), but to better-looking GC statistics (.count, .time, @alimans3 I feel you should also focus on gc.time.max statistics). Hence, this implementation would also a trade-off one might be interested in, with limited perf-impact.
One of the big question remains: how does the implementation with default IMemoryManager behave compared to current implementation: @alimans3 , can you provide figure from your JMH run with the default/dummy allocator, and with master code?

Sure, will be doing that later.

This regression is a result of the implementation of the pool which i think is far from perfect.

Could you post benchmarks with a pool implementation which outperforms the GC then please? This library is widely used and performance regressions can't be accepted for dubious benefit.

This regression is a result of the implementation of the pool which i think is far from perfect.

Could you post benchmarks with a pool implementation which outperforms the GC then please? This library is widely used and performance regressions can't be accepted for dubious benefit.

I am not for making this or any pool implementation default nor adding it as an option.
I am only for giving the users the option of doing their own thing.
The implementation is just for the sake of showing that it can be beneficial for some users to have this option. And the memory enhancements vs low perf impact could be a tradeoff someone is willing to take.

I understand the custom IMemorymanager leads to a 10% perf degradation (91 -> 100), but to better-looking GC statistics (.count, .time, @alimans3 I feel you should also focus on gc.time.max statistics). Hence, this implementation would enable a seemingly nice trade-off, with limited perf-impact.
One of the big question remains: how does the implementation with default IMemoryManager behave compared to current implementation: @alimans3 , can you provide figure from your JMH run with the default/dummy allocator, and with master code? (the point being to confirm/infirm default implementation is impact-less for the vast majority of library usage.)
after.log

I attached the results of the same benchmark run on master branch.
The results are similar in terms of performance.
For GC time, it seems it is not consistent but the GC count number is similar.

I don't think we should provide the custom allocator implementation, this is just to be able to demo.

This appeared legitimate to me at the beginning. However, such a custom allocator would help demonstrating the benefit of your proposition, and how the community would benefit from it. Else, it feels like we are complexifying the code for the benefit of your own project, with no immediate benefit to others (i.e. as we all agree writing+testing such a custom allocator is all but trivial).
Again, we may prefer focusing in make such a development easier (e.g. removing some final keywords, wrapping array allocations in overridables methods), but not injecting the whole concept of IAllocationManager in the core classes.

I had a try with your branch @alimans3 , forced the use of your custom allocator and ran tests: KO. I switch some Map into a ConcurrentHashMap to help thread-safety. But TestFastAggregation.testAndWithIterator (amongst others) seems to demonstrate some data corruption. Your custom allocator is buggy? Please demonstrate the whole thing can work, providing a full-fledged allocator. Its performance may be slightly worst, it is OK as long as the naive allocator would have no impact.
-> In addition to discussed performance elements, please provide a rock-solid PR, with a not naive allocator, to demonstrate the whole thing is achievable.
-> I believe we may not able to achieve a not_naive+not-very_slow+working_for_everybody allocator, but I feel you may achieve something matching your constrains and your executions pathes @alimans3

In addition to discussed performance elements, please provide a rock-solid PR, with a not naive allocator executed through all tests, to demonstrate the whole thing is achievable.

this is obviously achievable with a new test set, but it’s quite an undertaking for one simple reason: RoaringBitmap used not to be  closeable, so the tests don’t close the bitmaps.
My tests for my prototype indicated that there were use after free bugs because not all of the places a container should be closed (in thousands of lines of code written over many years) were identified, and that there were double closes, so there was an expectation on my part that if that work got picked up, the missing/double closes would be identified as a prerequisite to consideration to merging.

Okay so as a summary:
1- We need to provide a custom allocator to show the community the way to use it (TODO: Enhance allocator for thread-safety)
2- This is definitely not a rock-solid PR, it is just a demo of value, I will be adding tests, and validating all tests can run with the custom allocator (TODO: Validate all tests with new allocator)
3- We need to identify 2 scenarios:
Scenario A:
Missing free: harmless but we should find all places where this is the case
Scenario B:
Use after free: this is the buggy scenario, we need to find those places and I think running the tests with the custom allocator will help with that.
(TODO: Identify cases where these scenarios occur and fix them)
Before I go on with these TODOs, I think we need to have a consensus on whether we want to move forward with this option or not now that its potential is clear.
@richardstartin @blacelle @lemire let me know what you think.

I can maybe clarify the 'governance' part. I pretty much always take the same stance on these matters and I think it is a view that has wide support. Hopefully, you will approve too. Overall, we take a conservative stance. So if there is going to be disagreement over a pull request, then the pull request is not merged. I think that we all appreciate that you are producing high-quality work and that you have put a lot work on the current (draft) PR. You are obviously a capable programmer. I view your proposal as a research project. There is a lot of risk involved. The project cannot take this risk: you have to carry the burden of the proof. Sadly, I think it needs to be flushed out quite a bit before people can properly assess it.
So yes, this means that you could work for weeks, build a lot of code, produce a lot of benchmarks, and we could decide not to merge the PR. It follows because your proposal is risky.

this is obviously achievable with a new test set,

I feel this is not obvious while not breaking the not_very_slow constrain.

Yep. Hence, any way, maybe we should not spent this much energy into this PR until the whople work is done. We can try discouraging @alimans3 before we've done the work. Or let him do the work, and either accept this is not feasible in a for-all-users (as the simplest not naive allocator would give poor result), or come to us with a good implementation, still leaving options on the table (like keeping such a code in an extras module).

Missing free: harmless but we should find all places where this is the case

Not harmeless, as your PR invites people to use a custom allocator, and these people would encounter memory-leaks/bugs.

Before I go on with these TODOs, I think we need to have a consensus on whether we want to move forward with this option or not now that its potential is clear.

The consensus seems to be we're not convinced, for various reasons.

I am going to close this. The consensus is that it is not yet ready.


Add Enzyme v0.12 to compat

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.89%. Comparing base (3caef5e) to head (7645eb8).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #193      +/-   ##
==========================================
- Coverage   96.75%   95.89%   -0.87%
==========================================
  Files          67       15      -52
  Lines        3364     1560    -1804
==========================================
- Hits         3255     1496    -1759
+ Misses        109       64      -45

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Updated links in Prerequisites section

@mtarng updating links in service management readme

Hi @yayitserica I believe the current links are now up to date and should work; they were updated in https://github.com/microsoft/bedrock/pull/1363, but please let me know if I've missed something.

Links are fixed, removing this branch


"extract-text-webpack-plugin" loader is used without the corresponding plugin

Hi,
my production builds suddenly stopped to work. I haven done any updates or something and ended up with below error. I'm using Angular 5 having webpack 3.6.0, production build with AOT.
ERROR in Module build failed: Error: "extract-text-webpack-plugin" loader is used without the corresponding plugin, refer to https://github.com/webpack/extract-text-webpack-plugin for the usage example at Object.pitch (/Users/.../node_modules/extract-text-webpack-plugin/dist/loader.js:53:11)
Here is my production config file:
const ExtractTextPlugin = require('extract-text-webpack-plugin');
const CopyWebpackPlugin = require('copy-webpack-plugin');
const webpack = require('webpack');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const CleanWebpackPlugin = require('clean-webpack-plugin');
const path = require('path');
const ngToolsWebpack = require('@ngtools/webpack');

const ENV = process.env.NODE_ENV = process.env.ENV = 'prod';

module.exports = {
    entry: {
        'polyfills': './polyfills.ts',
        'vendor': './vendor.ts',
        'app': './app/main-aot.ts'
    },
    resolve: {
        extensions: ['.ts', '.js', '.json']
    },
    output: {
        path: path.join(__dirname, 'wwwroot'),
        filename: 'js/[name].[hash:6].bundle.js',
        chunkFilename: 'js/[id].[hash:6].chunk.js'
    },
    module: {
        rules: [
            {
                test: /\.ts$/,
                use: '@ngtools/webpack'
            },
            {
                test: /\.html$/,
                use: 'html-loader'
            },
            // {
            //     test: /\.(png|jpg|gif|ico|woff|woff2|ttf|svg|eot)$/,
            //     use: 'file-loader?name=assets/[name]-[hash:6].[ext]',
            // },
            {
                test: /\.(png|jpe?g|gif|ico)$/,
                use: 'file-loader?name=assets/[name].[ext]',
            },
            { test: /\.(woff2?|ttf|eot|svg)$/,
                use: 'url-loader'
            },
            {
                test: /\.css$/,
                exclude: path.resolve(__dirname, "app"),
                use: ExtractTextPlugin.extract({
                    fallback: 'style-loader', use: [
                        { loader: 'css-loader', options: { minimize: true } }
                    ]
                })
            },
            {
                test: /\.css$/,
                include: path.resolve(__dirname, "app"),
                use: 'raw-loader'
            }
        ]
    },
    plugins: [
        // AoT plugin.
        new ngToolsWebpack.AngularCompilerPlugin({
            tsConfigPath: './tsconfig-aot.json'
        }),
        new webpack.optimize.ModuleConcatenationPlugin(),
        new webpack.optimize.CommonsChunkPlugin({
            name: ['app', 'vendor', 'polyfills']
        }),
        new CleanWebpackPlugin(
            [
                './wwwroot/js/',
                './wwwroot/css/',
                './wwwroot/assets/',
                './wwwroot/index.html'
            ]
        ),
        new CopyWebpackPlugin([
            { from: './assets', to: './assets' },
        ]),
        // inject in index.html
        new HtmlWebpackPlugin({
            template: './index.html',
            inject: 'body',
            filename: 'index.html'
        }),
        new ExtractTextPlugin('css/[name]-[hash:6].bundle.css'),
        new webpack.DefinePlugin({
            'process.env': {
                'ENV': JSON.stringify('prod')
            }
        }),
        new webpack.optimize.UglifyJsPlugin({
            compress: {
                warnings: false
            },
            output: {
                comments: false
            },
            sourceMap: false
        }),
        new webpack.ProvidePlugin({
            jQuery: 'jquery',
            $: 'jquery',
            jquery: 'jquery'
        })
    ]
};

I'm running the following command to build my project:
rm -rf .aot wwwroot node_modules/ && npm cache clean -f && npm install && npm run ngc && npm run build:dist
Also posting my package.json:
{
  "name": "myapp",
  "version": "1.2.3",
  "description": "myappdesc",
  "repository": {},
  "scripts": {
    "ngc": "ngc -p ./tsconfig-aot.json",
    "start": "concurrently \"webpack-dev-server --env=dev --hot --inline --host <IP_ADDRESS> --port 3000 --open\" \"dotnet run\" ",
    "build:dev": "webpack --env=dev --progress --profile --colors",
    "build:dist": "webpack --env=prod --progress --profile --colors",
    "lint": "tslint ./app/**/*.ts -t verbose",
    "tsc": "tsc",
    "tsc:w": "tsc -w"
  },
  "license": "MIT",
  "dependencies": {
    "@agm/core": "1.0.0-beta.3",
    "@angular/animations": "5.2.10",
    "@angular/common": "5.2.10",
    "@angular/compiler": "5.2.10",
    "@angular/core": "5.2.10",
    "@angular/forms": "5.2.10",
    "@angular/http": "5.2.10",
    "@angular/platform-browser": "5.2.10",
    "@angular/platform-browser-dynamic": "5.2.10",
    "@angular/router": "5.2.10",
    "@angular/upgrade": "5.2.10",
    "@angularclass/hmr": "2.1.3",
    "@swimlane/ngx-charts": "7.4.0",
    "@types/jquery": "^3.2.17",
    "angular-calendar": "^0.24.0",
    "angular2-moment": "^1.8.0",
    "angular2-text-mask": "^8.0.4",
    "angular2-uuid": "^1.1.1",
    "angularfire2": "5.0.0-rc.6.0",
    "angulartics2": "5.3.0",
    "bootstrap": "^3.3.7",
    "core-js": "^2.4.1",
    "firebase": "4.12.1",
    "font-awesome": "^4.7.0",
    "ie-shim": "~0.1.0",
    "jquery": "^3.2.1",
    "moment": "^2.18.1",
    "moment-timezone": "^0.5.14",
    "ng2-file-upload": "1.3.0",
    "ng2-select": "2.0.0",
    "ngx-bootstrap": "2.0.3",
    "ngx-infinite-scroll": "^0.5.1",
    "ngx-slick": "^0.1.3",
    "ngx-toastr": "8.4.0",
    "ngx-uploader": "^4.3.0",
    "promise-polyfill": "^6.0.2",
    "reflect-metadata": "^0.1.8",
    "rxjs": "5.5.2",
    "slick-carousel": "1.8.1",
    "text-mask-core": "^5.0.1",
    "traceur": "^0.0.96",
    "zone.js": "^0.8.4"
  },
  "devDependencies": {
    "@angular/compiler-cli": "5.2.10",
    "@ngtools/webpack": "1.10.2",
    "@types/core-js": "^0.9.34",
    "@types/googlemaps": "^3.26.1",
    "@types/jasmine": "^2.5.54",
    "@types/moment-timezone": "^0.5.3",
    "@types/node": "8.0.26",
    "angular-router-loader": "^0.6.0",
    "angular2-template-loader": "^0.6.2",
    "awesome-typescript-loader": "^3.2.3",
    "canonical-path": "0.0.2",
    "clean-webpack-plugin": "^0.1.16",
    "codelyzer": "^3.1.2",
    "concurrently": "^3.5.0",
    "copy-webpack-plugin": "^4.2.0",
    "css-loader": "^0.28.7",
    "css-to-string-loader": "^0.1.3",
    "extract-loader": "^1.0.1",
    "mini-css-extract-plugin": "^0.4.4",
    "extract-text-webpack-plugin": "^3.0.2",
    "file-loader": "^1.1.5",
    "html-loader": "^0.5.1",
    "html-webpack-plugin": "^2.30.1",
    "http-server": "^0.10.0",
    "json-loader": "^0.5.7",
    "lite-server": "^2.3.0",
    "lodash": "^4.17.4",
    "node-sass": "^4.5.3",
    "protractor": "~5.1.2",
    "raw-loader": "^0.5.1",
    "rimraf": "^2.6.1",
    "sass-loader": "^6.0.6",
    "source-map-loader": "^0.2.1",
    "style-loader": "^0.18.2",
    "to-string-loader": "^1.1.5",
    "ts-helpers": "^1.1.2",
    "tslint": "^5.7.0",
    "tslint-loader": "^3.5.3",
    "typescript": "2.8.3",
    "typings": "^2.1.1",
    "url-loader": "^0.5.9",
    "webpack": "^3.6.0",
    "webpack-dev-server": "^2.9.1"
  },
  "-vs-binding": {
    "BeforeBuild": [
      "build:dev"
    ]
  }
}

I'm really hopeless because this started on our PROD environment and we can't go out with update / hotfix.
Thanks.

It looks like there are some others with this problem in the closed issues:
Closed issues: 'loader is used without the corresponding plugin'
This issue in the angular-cli repo also looked very similar:
https://github.com/angular/angular-cli/issues/5628
Hope that helps.

DEPRECATED
Please use: https://github.com/webpack-contrib/mini-css-extract-plugin


Fix README typo Typescript->Typeshare

Oops just noticed a PR for this already exists, so closing. Sorry for the noise!


pipenv results with "ModuleNotFoundError: No module named"; but venv works

Hi,
While trying to utilize pipenv for python package management,
on OSX 10.14.6, I suspect I've got tangled with resolving installed site packages,
First is the output of     $ pipenv --support | pbcopy -
$ pipenv --support
Pipenv version: '2018.11.26'
Pipenv location: '/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv'
Python location: '/usr/local/Cellar/pipenv/2018.11.26_2/libexec/bin/python3.7'
Python installations found:

3.7.4: /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python3
3.7.4: /usr/local/bin/python3
3.7.4: /usr/local/bin/python3.7m
2.7.10: /usr/bin/python
2.7.10: /usr/bin/pythonw
2.7.10: /usr/bin/python2.7

PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.7.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '18.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT '
                     '2019; root:xnu-4903.270.47~4/RELEASE_X86_64',
 'python_full_version': '3.7.4',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

PATH
TERM_PROGRAM
SHELL
TERM
PIP_PYTHON_PATH
TMPDIR
Apple_PubSub_Socket_Render
TERM_PROGRAM_VERSION
ANDROID_SDK_ROOT
TERM_SESSION_ID
JAVA_OPTS
USER
SSH_AUTH_SOCK
__CF_USER_TEXT_ENCODING
VIRTUAL_ENV
TMUX
PIPENV_ACTIVE
_
PWD
JAVA_HOME
ITERM_PROFILE
TMUX_PANE
XPC_FLAGS
PYTHONDONTWRITEBYTECODE
XPC_SERVICE_NAME
COLORFGBG
HOME
SHLVL
ITERM_SESSION_ID
LOGNAME
PIP_DISABLE_PIP_VERSION_CHECK
LC_CTYPE
COLORTERM
PIP_SHIMS_BASE_MODULE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

PIPENV_ACTIVE: 1

Debug–specific environment variables:

PATH: /usr/local/Cellar/pipenv/2018.11.26_2/libexec/tools:/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin:/usr/local/Cellar/pipenv/2018.11.26_2/libexec/tools:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin
SHELL: /bin/bash
PWD: /Users/elia/BE_Blog_posts/server_api
VIRTUAL_ENV: /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P

Contents of Pipfile ('/Users/elia/BE_Blog_posts/server_api/Pipfile'):
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
flask = "*"

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/Users/elia/BE_Blog_posts/server_api/Pipfile.lock'):
{
    "_meta": {
        "hash": {
            "sha256": "a82b674d67d29678775ff6b773de1686a9593749ec14483b0d8e05131b662286"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "click": {
            "hashes": [
                "sha256:2335065e6395b9e67ca716de5f7526736bfa6ceead690adf616d925bdc622b13",
                "sha256:5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7"
            ],
            "version": "==7.0"
        },
        "flask": {
            "hashes": [
                "sha256:13f9f196f330c7c2c5d7a5cf91af894110ca0215ac051b5844701f2bfd934d52",
                "sha256:45eb5a6fd193d6cf7e0cf5d8a5b31f83d5faae0293695626f539a823e93b13f6"
            ],
            "index": "pypi",
            "version": "==1.1.1"
        },
        "itsdangerous": {
            "hashes": [
                "sha256:321b033d07f2a4136d3ec762eac9f16a10ccd60f53c0c91af90217ace7ba1f19",
                "sha256:b12271b2047cb23eeb98c8b5622e2e5c5e9abd9784a153e9d8ef9cb4dd09d749"
            ],
            "version": "==1.1.0"
        },
        "jinja2": {
            "hashes": [
                "sha256:065c4f02ebe7f7cf559e49ee5a95fb800a9e4528727aec6f24402a5374c65013",
                "sha256:14dd6caf1527abb21f08f86c784eac40853ba93edb79552aa1e4b8aef1b61c7b"
            ],
            "version": "==2.10.1"
        },
        "markupsafe": {
            "hashes": [
                "sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473",
                "sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161",
                "sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235",
                "sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5",
                "sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff",
                "sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b",
                "sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1",
                "sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e",
                "sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183",
                "sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66",
                "sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1",
                "sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1",
                "sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e",
                "sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b",
                "sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905",
                "sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735",
                "sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d",
                "sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e",
                "sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d",
                "sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c",
                "sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21",
                "sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2",
                "sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5",
                "sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b",
                "sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6",
                "sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f",
                "sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f",
                "sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7"
            ],
            "version": "==1.1.1"
        },
        "werkzeug": {
            "hashes": [
                "sha256:87ae4e5b5366da2347eb3116c0e6c681a0e939a33b2805e2c0cbd282664932c4",
                "sha256:a13b74dd3c45f758d4ebdb224be8f1ab8ef58b3c0ffc1783a8c7d9f4f50227e6"
            ],
            "version": "==0.15.5"
        }
    },
    "develop": {}
}

The question is how can I fix pipenv so I will be able to successfully manage packages with pipenv instead of pythons working venv
The full story -

installed pipenv via brew -> as instructed in https://docs.pipenv.org/en/latest/install/#installing-pipenv
The pip file contect -

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
flask = "*"
flask-cors = "*"
flask-pymongo = "*"
pymongo = "*"
bson = "*"

[requires]
python_version = "3.7"

First before running pipenv install; deleting any preious using pipenv --rm and cleaning,
and followed clearing cache dirs as explained in: https://docs.pipenv.org/en/latest/diagnose/#your-dependencies-could-not-be-resolved
running pipenv install - finishes with no errors

Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
Creating a virtualenv for this project…
Pipfile: /Users/elia/BE_Blog_posts/server_api/Pipfile
Using /usr/local/bin/python3 (3.7.4) to create virtualenv…
⠹ Creating virtual environment...Using base prefix '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python3.7
Also creating executable in /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python
Please make sure you remove any previous custom paths from your /Users/elia/.pydistutils.cfg file.
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/local/bin/python3

✔ Successfully created virtual environment!
Virtualenv location: /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (6b5575)!
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 12/12 — 00:00:02

starting pipenv and attempting to run the app resolves in failure for ModuleNotFoundError: No module named 'flask'

python app.py
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    from flask import Flask
ModuleNotFoundError: No module named 'flask'

The Bellow is from within pipenv shell -

which python
/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python

python -m site
sys.path = [
    '/Users/elia/BE_Blog_posts/server_api',
    '/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/lib/python37.zip',
    '/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/lib/python3.7',
    '/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/lib/python3.7/lib-dynload',
    '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7',
    '/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/lib/python3.7/site-packages',
]
USER_BASE: '/Users/elia/.local' (exists)
USER_SITE: '/Users/elia/.local/lib/python3.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

pip list
Package          Version
---------------- ----------
bson             0.5.8
certifi          2018.10.15
Click            7.0
Flask            1.1.1
Flask-Cors       3.0.8
Flask-PyMongo    2.3.0
itsdangerous     1.1.0
Jinja2           2.10.1
MarkupSafe       1.1.1
pip              19.1.1
pipenv           2018.11.26
pymongo          3.8.0
python-dateutil  2.8.0
setuptools       41.0.1
six              1.12.0
virtualenv       16.3.0
virtualenv-clone 0.3.0
Werkzeug         0.15.5
wheel            0.33.4

After not being able to resolve this lack of ability to load packages, I've tried the Pythons default venv following: https://opensource.com/article/19/4/managing-python-packages
and after the activation of venv I'm able to successfully start the app and non modules are missing -

### output is when venv is activated (sourced activae script)
$ which python
/Users/elia/BE_Blog_posts/server_api/be_test1/bin/python

$ which pip
/Users/elia/BE_Blog_posts/server_api/be_test1/bin/pip

$ pip list
Package       Version
------------- -------
Click         7.0
Flask         1.1.1
Flask-Cors    3.0.8
Flask-PyMongo 2.3.0
itsdangerous  1.1.0
Jinja2        2.10.1
MarkupSafe    1.1.1
pip           19.2.1
pymongo       3.8.0
setuptools    40.8.0
six           1.12.0
Werkzeug      0.15.5

# python loads flask package and runs as expected -
$ python app.py
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://<IP_ADDRESS>:5000/ (Press CTRL+C to quit)

I would appreciate any help with troubleshooting and fixing lovely pipenv,
Thanks,
Elia.

starting pipenv and attempting to run the app resolves in failure for ModuleNotFoundError: No module named 'flask'

Just to clarify a fact that you didn't mention it: did you activate the shell before this point?
Within pipenv shell, can you run the following two commands?
$ pip --version
$ python -c "from flask import Flask"

Thank you @frostming for the quick response,
Valid question, this is the first time I'm utilising pipenv on this machine,
I've double checked that even with the previous path of the project, before I copied the files over to another dir, pipenv shell generates same issue of ModuleNotFoundError,
does it help to know that I can run the app using the "global" python libs, meaning outside of pipenv, without starting the pipenv shell.
Additional requested output - from within pipenv shell
$ pip --version
pip 19.1.1 from /usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pip (python 3.7)
$  python -c "from flask import Flask"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'flask'

Again thank you very very much for your time and kind assistance.

$ pip --version
pip 19.1.1 from /usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pip (python 3.7)

Wrong python location. Do you tangle the PATH somewhere? Also, echo $PATH to see what is happening.

within pipenv shell -
$ echo $PATH
/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin:/usr/local/Cellar/pipenv/2018.11.26_2/libexec/tools:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin

I've tried to avoid tempering with my PATH, but those looking at my history commands I see my attempts to probably follow some HowTo google-ing how to fix this issue -
$ grep -i "PATH" ~/.bash*
/Users/elia/.bash_history:export PATH="$PATH:/Users/elia/.local"
/Users/elia/.bash_history:echo $PATH
/Users/elia/.bash_history:PATH=$PATH:/Users/elia/bin
/Users/elia/.bash_history:echo $PATH

no PATH settings in .bashrc /.profile files as well,
Path outside pipenv shell:
$echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin

during the execution of pipenv install, there is a reference to the file: ~/.pydistutils.cfg,
I'm not familiar with this file, but bellow it's content -
$  cat /Users/elia/.pydistutils.cfg

[global]
verbose=1

[install]
install-scripts=$HOME/bin

[easy_install]
install-scripts=$HOME/bin

and I have 2 aliases set for python, in ~/.bashrc -
alias ipython='python -m IPython'
alias ipython3='python3 -m IPython'
besides that, no python/PATH tempering as far as I remember.

$ echo $PATH
/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin:/usr/local/Cellar/pipenv/2018.11.26_2/libexec/tools:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin

The path seems correct, but why did python points to the interpreter outside the venv? What if you remove ~/.pydistutils.cfg?

I've tried to remove ~/.pydistutils.cfg,
remember doing it, and it did not help, will double check,
What about brew installing python3 incorrectly? as I remember osx did not come with python3 and I've installed it via homebrew,
can this somehow mess the python env/site packages loading?
Looking at the interpreter link from within venv -
$ which python
/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python
## and here I was surprised by not seeing the same path.
$  whereis python
/usr/bin/python

$ ll /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python   lrwxr-xr-x  1 elia  staff     9B Aug  8 13:28 /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python -> python3.7

$  ll /usr/bin/python
-rwxr-xr-x  1 root  wheel    65K May  4 08:25 /usr/bin/python

Need more info:
$ which pip
$ /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python -m pip list
$ /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/python -m pip --version

Came back withvery good news, I've performed the following -

removing ~/.pydistutils.cfg,
pipenv --rm
rm Pipfile.lock
pipenv install
Everythong works -

pipenv run python app.py
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://<IP_ADDRESS>:5000/ (Press CTRL+C to quit)

2 days I've been running pipenv --rm, delete cahce, install, completely reinstall pipenv, remove ~/.pydistutils.cfg,
probably missed the above...
from within the working pipenv shell -
pipenv run which pip
/Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/bin/pip
$  pipenv run pip --version
pip 19.2.1 from /Users/elia/.local/share/virtualenvs/server_api-gYoqao8P/lib/python3.7/site-packages/pip (python 3.7)

$  pipenv run pip list
Package         Version
--------------- -------
bson            0.5.8
Click           7.0
Flask           1.1.1
Flask-Cors      3.0.8
Flask-PyMongo   2.3.0
itsdangerous    1.1.0
Jinja2          2.10.1
MarkupSafe      1.1.1
pip             19.2.1
pymongo         3.8.0
python-dateutil 2.8.0
setuptools      41.0.1
six             1.12.0
Werkzeug        0.15.5
wheel           0.33.4

# all dependency packages install, and python loads modules and executes as expected.

Any additional info that can assist in preventing the next similar poor osx/python env/pipenv newcomer getting tangled with 101 pipenv? I've tried to follow the documentation instructions step by step(brew install pipenv), by somehow got into the above and could not resolve this without your help..
Again thank you very much 👍

Below is a common path to solving ModuleNotFound error:

Check whether the desired package is installed by pip list.
Check which pip is being used.
Check all possibilities to have the installation destination changed:

PYTHONHOME envar.
pip.conf
The old school .pydistutils.cfg
env|grep -i PIP


[code] does not set a monospace font as expected

The code that parses the [code] tag when displaying the in-editor help claims to be setting the font to a monospaced one [1], but it does not appear to be working.
See e.g., after adding the tag in this method:

The output in the editor:

No particular formatting is visible.
[1] https://github.com/godotengine/godot/blob/master/tools/editor/editor_help.cpp#L1204-L1209

Just to be sure, I tested with just one word within the [code]/[/code] tags, but the result is the same.

:+1:, please fix :smile:

yeah this is a mess, I'm going to add variable size fonts soon to attempt
to solve the problem
On Sun, Feb 28, 2016 at 12:11 PM, Bojidar Marinov<EMAIL_ADDRESS>wrote:

[image: :+1:], please fix [image: :smile:]
—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/issues/3750#issuecomment-189889463.

DynamicFont was already added. What's the progress on this?

It works fine for both [code] and [codeblock] in the in-editor help now. ([codeblock] is still not display in a nice way, but this very bug is fixed).


Not able to insert data into ES using elasticsearch-hadoop-2.1.0.Beta4.jar

Would appreciate if you could look into this issue.
I'm trying to insert data from hive to ES using elasticsearch-hadoop-2.1.0.Beta4.jar, but it's not successful. I was using elasticsearch-hadoop-2.0.2.jar version, it didn't work. That's why I downloaded elasticsearch-hadoop-2.1.0.Beta4.jar to try.  I can create external tables and read data from ES without any issues. So I think the jar is in the right directory(In my case, I put the jar file in /hive/lib folder).
My environment is :
Hadoop 2.0.0-cdh4.5.0
hive-common-0.10.0
My script is :
DROP TABLE IF EXISTS es_edge_v_security_price_rolling;

CREATE EXTERNAL TABLE es_edge_v_security_price_rolling (
    currency    STRING,
    data_date   STRING,
    price       DOUBLE,
sls_sec_sec_num STRING
    )
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'system/service', 'es.nodes' = 'localhost:9200');

INSERT OVERWRITE TABLE dod.es_edge_v_security_price_rolling
select e.currency, e.data_date, e.price, e.sls_sec_sec_num
from dod.edge_v_security_price_rolling e limit 10;

Error message from the failed reduce task log:
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:373)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:377)
at org.apache.hadoop.hive.ql.exec.LimitOperator.initializeOp(LimitOperator.java:41)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
... 14 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:315)
... 25 more
2015-06-12 05:18:10,301 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

Please let me know if you need more information. Thanks.

edited your post to add minor formatting - please do so yourself in the future
@xiaoyi78 It looks like there's a problem with your Hadoop install.
The NPE stacktrace is triggered by the Hadoop classes in particular ReflectionUtils.setJobConf which tries to do some reflection:
    //If JobConf and JobConfigurable are in classpath, AND
    //theObject is of type JobConfigurable AND
    //conf is of type JobConf then
    //invoke configure on theObject

This happens way before the connector kicks in and suggests that you might be having an incomplete Hadoop/Hive configuration or classpath or potentially you are using a mixture of Hadoop 1 and 2. Try using a different Hadoop install just for testing or upgrade your CDH install to the latest stable branch (even on 4.5.x).

Thanks for your quick response and  pointing out this is not related to the connector. We will investigate further on our configuration. Cheers.

Sorry Costin, the error message I provided to you in my first post is from 'syslog' tab, but when I looked into 'stderr' tab, it has the error message related to elastic search. Please see below. Sorry, I need to learn how to format the code block below. Please bear with me one more time. It seems to me I got some classpath issue. But I still don't understand what went wrong for my classpath, I can create external table pointing to ES and read data from it. It would fail if my classpath was not set up properly. Appreciate if you could shed some light...
The code block below :
Continuing ...
java.lang.ClassNotFoundException: org.elasticsearch.hadoop.hive.EsHiveOutputFormat
Continuing ...
java.lang.NullPointerException
Continuing ...
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:315)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:377)
at org.apache.hadoop.hive.ql.exec.LimitOperator.initializeOp(LimitOperator.java:41)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)

Sorry, just reformat my log block for my previous post.
Continuing ...
java.lang.ClassNotFoundException: org.elasticsearch.hadoop.hive.EsHiveOutputFormat
Continuing ...
java.lang.NullPointerException
Continuing ...
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:315)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:377)
at org.apache.hadoop.hive.ql.exec.LimitOperator.initializeOp(LimitOperator.java:41)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)

@xiaoyi78 the es-hadoop jar is not found in the classpath hence the CNFE. Hive is notorious for being hard to setup the classpath; it's best to double check their documentation (here is one link).
One reliable configuration for the latest versions of Hive is setting the HIVE_AUX_JARS_PATH as apparently setting the hive.xml is ignored by HiveServer2.
Note that this is not a es-hadoop bug - its classes are simply not found and thus cannot be loaded by Hive.

@xiaoyi78 By the way, make sure you only use just one version of es-hadoop. If you are using multiple (1.3 and 2.0 and 2.1), the classes are likely to trip on each other as the runtime will pick classes from each jar and these are not compatible.

Thanks  Cosin. Finally, I figured out the issue. As you mentioned in another post, most of this type issue was caused by classpath not set up correctly. In my case, I had to add the es-hadoop jar file into another location -- "CDH-4.5.0-1.cdh4.5.0.p0.30/lib/hadoop-0.20-mapreduce/lib", which is for map-reduce job. This makes sense because the exception was thrown during reduce task and it could not find es-hadoop jar file.
So for people who experience the similar issue,  please note apart from setting up the correct Hive classpath, but also for MapReduce job classpath.
Thanks again for the support.